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ABSTRACT 


This  thesis  examines  validity  issues  associated  with  the 
use  of  data  collection  techniques  in  information  systems 
research.  It  presents  an  analysis  of  37  studies  that 
purported  to  empirically  assess  the  effectiveness  of 
information  systems.  These  studies  were  evaluated  to 
determine  the  validity  of  measures  of  effectiveness  of 
information  systems .  Each  study  was  reviewed  to  identify 
(1)  data  collection  techniques  used,  (2)  purported  measures 
of  the  techniqpies,  (3)  ways  in  which  the  techniques  were 
administered,  and  (4)  discussions  of  validity  issues  arising 
from  the  use  of  the  techniques . 

Findings  indicate  that  information  systems  researchers 
have  adopted  data  collection  techniques  commonly  used  by 
social  scientists;  however,  they  largely  ignore  or  are 
unaware  of  associated  validity  issues .  Over  three-quarter 
of  the  studies  involved  questionnaires  and  fewer  than  a 
quarter  addressed  validity  issues.  Consequently,  the 
credibility  of  information  systems  research  is  vulnerad^le  to 
challenge . 
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I .  INTRODUCTION 

A.  THE  MOTIVATION  TO  ACCURATELY  MEASURE  THE  EFFECTIVENESS 

OF  INFORMATION  SYSTEMS 

After  spending  billions  of  dollars  on  automating  the 
offices  of  white  collar  workers,  organizations  find  that 
they  must  scrutinize  the  claims  by  computer  vendors  about 
increased  productivity.  To  prudently  manage  a  resource  an 
organization  must  have  a  means  to  measure  its  benefits  as 
well  as  its  costs. 

An  appropriate  managerial  question  that  arises  is 
whether  an  organization  is  getting  a  return  on  its 
investment  in  a  resource.  This  necessity  is  not  diminished 
because  the  resource  in  question  is  an  information  system. 
Managers  of  information  systems  are  pressed  by  senior  line 
management  to  answer,  again  and  again,  the  question,  "What 
is  the  value  of  the  company's  investment  in  information 
systems?"  (Matlin,  1979) . 

According  to  Strassman  (1976) ,  "Industries  and 
governments  all  over  the  world  are  currently  struggling  to 
contain  rising  administrative  and  clerical  overheads  by 
automating  information  handling  in  the  office." 

While  the  clerical  work  force  has  been  the  fastest 
growing  segment  of  the  labor  force  in  nearly  every 
industrialized  country  since  the  1960s,  the  productivity  of 
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this  sector  has  been  flat  since  the  early  1970s  (Harris, 
1987) ,  This  has  been  the  result  despite  the  fact  that  40% 
of  capital  spending  (in  the  United  States)  has  been  directed 
at  information  technology. 

General  managers  in  the  United  States,  when  surveyed  by 
the  University  of  Minnesota  Management  Information  Systems 
Research  Center,  ranked  the  measuring  of  system 
effectiveness  as  the  fourth  most  critical  problem  they  face 
in  managing  information  systems  (Brancheau  and  Wetherbe, 
1987) .  Sixty  percent  of  the  management  information  systems 
executives  polled  in  the  Scune  survey  placed  the  measuring  of 
effectiveness  in  their  "top  ten"  list  of  critical  problems. 
The  assessment  of  system  success  has  been  in  the  top  ten 
list  of  critical  management  information  systems  problems 
since  it  was  first  formulated  in  1980. 

Information  systems  practitioners  are  clearly  expressing 
considerable  concern  over  the  effectiveness  of  information 
systems.  Consequently,  researchers  are  devoting 
considerable  efforts  to  studying  effectiveness  of  systems. 
Professional  journals  routinely  pxoblish  articles  reporting 
researchers'  findings  concerning  effectiveness  of 
infoirmation  systems;  however,  are  these  findings  based  upon 
measures  that  have  demonstrated  validity? 

B.  ARE  RESEARCHERS  ADDRESSING  VALIDITY  ISSUES? 

Evaluations  regarding  the  effectiveness  of  information 
systems  are  based  upon  measures  collected  and  analyzed  by 
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information  systems  researchers.  Such  measures  are  gathered 
using  a  variety  of  data  collection  techniques .  Researchers 
in  sociology,  such  as  Campbell  and  Stanley,  Denzin,  Phillips 
and  Suchman,  have  demonstrated  that  data  collection 
techniques  significantly  influence  data  validity. 

C.  THE  RESEARCH  QUESTION 

This  study  asks  if  information  systems  researchers,  who 
reported  measures  of  information  systems  effectiveness, 
sufficiently  addressed  validity  issues  arising  from  data 
collection  techniques. 

Subsidiary  research  questions  follow: 

1.  What  data  collection  instruments  do  researchers  use? 

2 .  What  do  researchers  purport  to  measure  with  the 
instruments? 

3.  How  do  researchers  administer  the  instruments? 

4 .  How  and  to  what  extent  do  researchers  address  validity 
issues? 

Using  these  research  questions,  the  study  will  assess  the 
general  awareness  and  sensitivity  of  researchers  to  validity 
issues . 

D.  SCOPE  or  THE  STUDY 

This  study  consists  of  an  einalysis  of  published 
empirical  methods,  since  1970,  which  were  used  to  evaluate 
effectiveness  of  information  systems.  The  analysis 
identifies  how  researchers  collected  data  when  evaluating 
the  effectiveness  of  information  systems  and  how  they 
subsequently  addressed  validity  issues .  Using  literature 
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from  sociology,  the  data  collection  techniques  used  by 
information  systems  researchers  are  exeunined  and  their 
impact  upon  data  validity  is  discussed. 

E .  ASSUMPTIONS 

It  is  assximed  that  the  37  articles  analyzed  in  this 
study  represent  the  current  state  of  information  systems 
research.  Findings  regarding  the  researchers  who  wrote 
these  articles  are  assumed  to  apply  to  information  systems 
researchers  in  general . 

F.  ORGANIZATION  OF  THE  STUDY 

This  study  is  organized  into  introduction,  literature 
review,  methodology,  findings,  and  conclusions  chapters. 

The  literature  review  chapter  explains  the  concept  of 
validity,  drawing  primarily  upon  works  of  prominent  social 
scientists .  The  methodology  chapter  details  the  selection 
of  studies  for  review  and  the  methods  in  which  these  studies 
were  analyzed.  The  findings  chapter  presents  data  collected 
during  the  analysis  of  37  articles  regarding  effectiveness 
of  systems .  The  conclusions  chapter  discusses  the 
significance  of  key  findings  and  provides  recommendations 
for  enhancing  the  credibility  of  information  systems 
research . 
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II.  LITERATURE  REVIEW 


A.  CLASSICAL  DEFINITION  OF  MEASUREMENT 

Stevens  (1951)  provides  a  popular  definition  of 
measurement,  "Measurement  is  the  assignment  of  numbers  to 
objects  or  events  according  to  rules."  This  definition  is 
appropriate  for  the  physical  sciences,  where  the  objects  and 
events  are  the  focus  of  research.  Information  systems 
researchers  often  measure  phenomena  which  are  typically  too 
a±istract  for  characterization  as  either  objects  or  events . 
Information  systems  researchers  are  currently  studying 
issues  of  effectiveness,  benefit,  and  success  of  information 
systems.  All  of  these  issues  are  too  abstract  for  direct 
observation  and  quantification. 

B.  MEASUREMENT  AND  THE  SOCIAL  SCIENCES 

Social  science  researchers  study  phenomena  which  are 

equally  abstract .  They  have  adopted  an  alternative 

definition  of  measurement,  as  suggested  by  Blalock  (1968) , 

"Sociological  theorists  often  use  concepts  that  are 
formulated  at  rather  high  levels  of  abstraction.  These 
are  quite  different  from  the  variables  that  are  the 
stock-in-trade  of  empirical  sociologists.  .  .  .  The 

problem  of  bridging  the  gap  between  theory  and  research 
is  then  seen  as  one  of  measurement  error.” 

Carmines  and  Zeller  (1979)  view  measurement  most  usefully  as 

the  process  of  linking  aJostract  concepts  to  empirical 

indicators . 
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Carmines  and  Zeller's  definition  of  measurement  in 
social  science  research  identifies  a  process  that  involves 
both  theoretical  as  well  as  empirical  considerations.  They 
further  elaborate, 

"From  an  empirical  standpoint,  the  focus  is  on  the 
observable  response- -whether  it  takes  the  form  of  a  mark 
on  a  self-administered  questionnaire,  the  behavior 
recorded  in  an  observational  study,  or  the  answer  given 
to  an  interviewer.  Theoretically,  interest  lies  in  the 
underlying  unobservable  (and  directly  unmeasuradale) 
concept  that  is  represented  by  the  response .... 
Measurement  focuses  on  the  crucial  relationship  between 
the  empirically  grounded  indicator (s) — that  is,  the 
observeible  response — and  the  underlying  unobservable 
concept ( s ) . " 

C.  MHJLSUREMEMT  AMD  IMFOBMATIOM  SYSTEMS 

To  deal  with  the  abstract  nature  of  information  systems 
issues,  many  information  systems  researchers  are  using  the 
measurement  process  and  data  collection  techniques  commonly 
found  in  social  science.  Information  systems  researchers 
often  use  observable  response,  unobserved^le  concepts,  and 
linking  relationships  to  measure  information  systems 
phenomena.  Figure  1  illustrates  how  the  measurement  process 
used  by  many  information  systems  researchers  mirrors  the 
measurement  process  used  in  social  science.  Consequently, 
these  information  systems  researchers  have,  consciously  or 
unconsciously,  adopted  the  measurement  process  of  social 
science . 
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EnPIRICALLY  GROUNDED 

INDICATORS  and  OBSERVABLE  RESPONSE 


Observations 
Records  and  documents 


FIGURE  1.  Information  Systems'  Measurement  Process 

Social  science  researchers  have  long  recognized  that  the 
validity  of  the  measurement  process  is  critical  to  deriving 
useful  information  from  any  collected  data.  Since  the  same 
measurement  process  is  now  used  by  information  systems 
researchers,  they  are  now  logically  siibject  to  the  same 
validity  considerations .  Information  systems  researchers 
who  use  this  form  of  measurement  must  understand  the  concept 
of  validity. 

D.  CONCEPT  OF  VALIDITY 

Suchman  (1967)  discusses  validity,  reliability,  their 
relationship,  and  its  importance.  Validity  refers  to  the 
degree  to  which  any  measure  or  procedure  succeeds  in  doing 
what  it  is  purported  to  do.  Closely  related  to  validity  is 
the  concept  of  reliability.  Reliability  refers  to  the 
degree  to  which  any  measure  or  procedure  can  be  depended  on 
to  secure  consistent  results  upon  repeated  application. 
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Validity  and  reliability  are  interdependent.  There  can 
be  no  validity  without  reliaibility .  When  a  measure  is  shown 
to  be  valid,  its  relieJoility  is  taken  for  grauited.  However, 
one  can  have  high  reliaibility  without  validity.  The 
demonstration  of  a  measure's  reliability  does  not  establish 
its  validity. 

Validity  is  the  most  important  criterion  for  evaluating 
a  measuring  instrument .  Every  phase  of  the  evaluation 
process  is  influenced  by  its  validity.  Validity  concerns 
the  formulation  of  objectives,  the  collection  of  data,  and 
the  interpretation  of  findings.  The  validity  of  an 
evaluation  study  refers  not  only  to  the  validity  of  its 
specific  criteria  or  measures,  but  also  to  the  theory 
underlying  the  formulation  of  the  hypotheses  concerning  the 
relationship  of  the  activities  to  the  objective. 

S.  TYPES  OF  VALIDITY 

Phillips  (1971)  states  that  to  determine  the  validity  of 
a  measuring  instrument,  one  must  evaluate  it  against 
relevant  or  appropriate  criteria.  Social  science 
researchers  use  numerous  criteria  to  determine  the  validity 
of  measuring  instruments . 

The  following  are  types  of  validity  and  thus  used  as 
criteria  to  verify  the  validity  of  measuring  instruments: 

1 .  Content  validity  is  the  making  of  a  judgement  as  to 
the  representativeness  and  comprehensiveness  of  the 
measuring  instrument's  indicators  constituting  a 
measurement  (Phillips,  1971) .  Content  validity  refers 
to  the  adsility  of  the  instruments  to  measure  what  it 
purports  to  measure. 
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2.  Face  validity  is  an  aspect  of  content  validity.  It  is 
the  extent  to  which  an  instriament  seems  to  measure 
what  it  is  supposed  to  measure.  Face  validity  is  the 
obvious  significance  of  the  measure.  It  is  usually  a 
judgment  made  by  the  designer  of  an  instrument,  after 
its  completion  (Phillips,  1971) . 

3 .  Consensual  validity  is  a  face  validity  which  uses  a 
panel  of  experts  to  develop  a  consensus  concerning  the 
validity  of  a  measure  (Suchman,  1967) . 

4 .  Criterion  or  correlational  validity  correlates  a 
measure  with  another  indicator  that  one  "knows” 
measures  what  is  being  sought.  The  new  measure  is 
correlated  with  an  existing,  accepted  measure,  thus 
este±>lishing  the  validity  of  the  new  measure .  High 
correlation  of  two  measures  demonstrates  the  validity 
of  the  new  measures  (Suchman,  1967) . 

5.  Concurrent  validity  is  a  type  of  criterion  validity 
that  refers  to  the  correlation  of  a  measure  and  a 
criterion  at  the  saune  point  in  time.  For  example, 
correlating  data  gathered  through  exit  polling 
interviews  to  official  voting  records  (Carmines  and 
Zeller,  1979) . 

6.  Predictive  validity  is  another  type  of  criterion 
validity  that  refers  to  the  correlation  of  present 
measures  with  behavior  that  is  predicted  to  take  place 
in  the  future  (Suchman,  1967)  .  For  exaunple, 
correlating  the  results  of  an  undergraduate's  SAT 
scores  to  his  grade  point  average  at  graduation. 

7 .  Construct  validity  concerns  the  degree  to  which 
measures  conform  in  accordance  with  theoretical 
expectations .  Consequently,  a  theoretical  concept 
must  exist  for  predicting  and  analyzing  the  measures . 
Measures  are  considered  to  have  construct  validity  if 
they  are  consistent  with  theoretical  expectations.  If 
they  are  inconsistent  with  theoretical  expectations, 
then  it  is  concluded  that  the  measures  lack  validity 
for  that  particular  concept  (Carmines  and  Zeller, 

1979)  . 

8.  Convergent -discriminant  validity  is  the  expectation, 
based  on  a  priori  or  theoretical  grounds,  that 
measures  would  relate  to  some  other  variable  or 
variaJDles.  Convergent  validity  assesses  the  agreement 
between  different  methods  of  measuring  the  same 
construct.  Discriminant  validity  assesses  the 
difference  in  results  when  the  same  method  is  used  to 
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measure  supposedly  different  attributes  of  the  same 
object.  (Brewer  and  Collins,  1981) 

To  establish  the  validity  of  measurements,  social 
science  researchers  verify  that  the  measures  correlate  with 
at  least  one  of  these  types  of  validity.  High  correlation 
to  a  specific  type  of  validity  allows  the  assignment  of  that 
particular  type  of  validity  to  the  measurements.  Low 
correlation  to  any  of  the  above  types  of  validity  fails  to 
estcUDlish  the  validity  of  the  measurements.  Expert 
judgement  or  empirical  procedures  serve  as  the  basis  for 
establishing  correlation,  based  upon  the  type  of  validity 
claimed.  Cronbach's  coefficient  alpha,  Pearson's  product- 
moment  correlations,  auid  Spearman's  rank  correlations  are 
examples  of  statistical  calculations  typically  used  when 
establishing  correlation  through  empirical  procedures. 

F.  INTERNAL  AND  EXTERNAL  VALIDITY 

The  assignment  of  a  type  of  validity  to  a  given  set  of 
measurements,  by  virtue  of  the  existence  of  a  high  degree  of 
correlation  to  an  estedalished  type  of  validity,  does  not 
resolve  the  issue  of  whether  the  experimental  design 
accurately  measures  the  phenomena  of  interest.  Further,  it 
doesn't  establish  whether  the  study  findings  can  be 
legitimately  extended  beyond  the  sample.  These  issues  are 
directly  addressed  as  the  concepts  of  internal  and  external 
validity.  The  existence  of  internal  and  external  validity 
establishes  the  meaningfulness  of  an  experimental  design  far 
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more  than  does  the  assignment  of  a  type  of  validity  to  a 
given  measurement . 

Campbell  and  Stanley  (1966)  define  internal  validity  as 
dealing  with  the  issue  of  whether  the  measurements  are  due 
to  the  effect  of  a  particular  independent  vari2Q5le  or 
causally  related  to  the  dependent  variable  of  interest.  It 
asks  whether  a  difference  exists  at  all  in  any  given 
comparison  or  if  an  observed  difference  is  an  artifact  of 
the  observational  process .  Internal  validity  provides 
assurances  that  the  measurements  from  a  research  design 
result  from  its  identified  varieJbles .  When  the  measurements 
are  causally  related  to  the  dependent  variadjle  or  other 
factors  for  which  no  accounting  is  made  in  the  research 
design,  the  internal  validity  of  the  measurements  becomes 
confounded. 

C^unpbell  and  Stanley  (1966)  define  external  validity  as 
the  degree  to  which  findings  can  be  generalized  to  larger 
populations  or  settings.  One  uses  external  validity  to 
project  the  findings  from  a  experiment  beyond  its  seunple, 
onto  some  larger  population  of  interest. 

G.  IMPORTANCE  OF  INTERNAL  VALIDITY 

Internal  validity  is  more  important  than  external 
validity.  Measurements  from  experimental  designs  that  do 
not  have  internal  validity  are  considered  unreliable  and 
unrepresentative  of  the  phenomena  of  interest . 

Consequently,  it  is  impossible  to  extend  a  study's  findings 
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beyond  the  seunple  population.  Internal  and  external 
validity  are  interdependent.  Without  internal  validity,  an 
experimental  design  can  not  have  external  validity.  The 
existence  of  internal  validity  does  not  insure  that  the 
experimental  design  will  possess  external  validity. 

H.  FACTORS  THREATENING  INTERNAL  VALIDITY 

Campbell  and  Stanley  (1966)  identify  12  factors 
jeopardizing  the  internal  or  external  validity  of 
experimental  designs.  Relevant  to  internal  validity,  they 
present  eight  factors,  which  if  not  controlled  in  the 
research  design,  might  result  in  effects  that  obscure  the 
effect  of  the  experimental  stimulus.  Some  of  these  factors 
threatening  internal  validity  deal  with  spurious  conclusions 
based  upon  observations  made  before  and  after  the  effect  of 
an  intervening  experimental  variable.  Others  deal  with 
erroneous  conclusions  based  upon  supposed  differences 
between  comparison  groups .  The  following  factors 
potentially  threaten  the  internal  validity  of  experimental 
designs : 

1 .  History,  the  external  events  occurring  between  the 
pretest  measurements  and  post-test  measurements  in 
addition  to  the  experimental  variable. 

2 .  Maturation,  refers  to  the  process  of  change  within  the 
respondents  themselves  as  a  function  of  the  passage  of 
time . 

3.  Testing,  the  effects  resulting  from  having  taken 
earlier  tests,  commonly  referred  to  as  re-test 
sophistication . 

4.  Instrumentation,  the  changes  occurring  in  the 
calibration  of  the  measuring  instruments,  observers. 
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or  scorers  produces  changes  in  the  observed 
measurements . 

5.  Statistical  regression,  where  groups  have  been 
selected  on  the  basis  of  their  extreme  scores . 

Measures  of  such  groups  tend  to  regress  towards  the 
mean  on  svibsequent  observations. 

6.  Selection  biases,  is  the  differential  selection  of 
respondents  for  the  sample  population. 

7.  Experimental  mortality,  is  the  non-random  drop-out 
of  participants  from  the  experiment . 

8 .  Selection-maturation  interaction,  in  multi-group 
designs,  their  interaction  is  confounded  with  the 
effect  of  the  experimental  variadjle . 

I.  FACTORS  THREATENING  EXTERNAL  VALIDITY 

Caunpbell  and  Stanley  (1966)  presents  four  factors  that 
can  jeopardize  external  validity  or  representativeness.  If 
not  controlled  in  the  research  design,  these  factors  might 
result  in  the  inappropriate  generalization  of  study  results 
to  larger  populations.  These  factors  are  (1)  the  reactive 
or  interactive  effect  of  testing,  (2)  the  interaction 
effects  of  selection  biases  and  the  experimental  variad^le, 

(3)  reactive  effects  of  experimental  arrangements,  and  (4) 
multiple-treatment  inference. 

Bernstein  et  al .  (1975)  elaborated  on  the  works  of 

Campbell  and  Stanley  (1966)  and  Bracht  and  Glass  (1968) , 
regarding  factors  that  threaten  external  validity  of 
research  findings.  They  have  grouped  these  factors  into  the 
following  categories: 

1 .  Selection  effects  result  from  infeasibility  of  drawing 
a  probaibility  sample  from  the  target  population. 
Instead  the  researcher  must  sometimes  use  a  biased 
sample.  The  types  of  such  selection  biases,  in  order 
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of  the  degree  to  which  they  threaten  external 
validity,  are  (1)  self -select ion  by  the  respondents 
into  treatment  and  control  groups,  (2)  selection  by 
excellence,  (3)  selection  by  expedience,  and 
(4)  non-random  posttreatment  matching. 

2.  Measurement  effects  includes  the  effects  of 
unreliedDility  and  invalidity  of  measurement,  the 
reactivity  of  some  measures,  and  interactions 
between  measurement  and  other  variaJ^les . 

3.  Confounded  treatment  effects  may  arise  when  (1)  a  lack 
of  standard  treatments  are  applied  across  saimpling 
elements,  (2)  multiple  treatments  are  applied  across 
seunpling  elements,  or  (3)  individual  level  variables 
(attributes)  of  the  sampling  elements  interact  with 
the  treatments . 

4 .  Situational  effects  result  from  the  conduct  of  the 
experiment  in  a  particular  context  that  is  not 
representative  of  all  contexts  to  which  the  researcher 
may  wish  to  extend  a  generalization. 

5.  Effects  due  to  differential  mortality  result  from 
differential  siibject  and  program  losses  during 
social  experiments .  When  the  losses  are  different  for 
the  treatment  and  control  groups,  and  these  losses  are 
not  random,  then  external  validity  is  threatened. 

J.  DATA  COLLECTION  TBCXNIQDES 

Galtung  (1967)  states  that  all  social  science  data  are 

obtained  in  either  formal  (laboratory)  or  informal  (field) 

settings  and  involve  either  verbal  (oral  and  written)  or 

nonverbal  acts  or  responses.  The  combination  of  the  two 

settings  for  data  collection  and  two  ways  responses  are 

exhibited  have  led  to  the  development  of  these  commonly  used 

data  collection  techniques: 

1 .  Observations  of  behavior 

2 .  Experiments 

3 .  Reports  by  others 

4 .  Analyses  of  existing  records  and  documents 
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5. 


Interviews 


6.  Questionnaires. 

Each  data  collection  technique  has  its  own  characteristics 
and  resulting  impacts  upon  validity. 

1 .  Observations  of  Behavior 

There  are  primarily  three  forms  of  collecting 
observations  of  behavior; 

1 .  Simple  observation 

2 .  Contrived  observation 

3.  Participant  observation. 

Simple  observation  of  behavior  involves  an  observer  who  has 
no  control  over  the  behavior  or  situation  in  question,  and 
plays  an  vinobserved,  passive  emd  tanobtrusive  role  in  the 
research  situation  (Webb  et  al.,  1966).  Denzin  (1970) 
identifies  five  types  of  simple  observations;  exterior  body 
and  physical  signs,  expressive  movement,  physical  location 
analysis,  observation  of  language,  and  time  sampling 
analysis . 

Denzin  (1970)  identifies  sources  of  invalidity  with 
simple  observation.  Populations  change  over  time  and 
location,  making  it  very  difficult  to  gather  representative 
samples .  Small  samples  of  much  larger  populations  may 
result  in  quite  tentative  generalizations .  Observers  or 
measuring  devices  make  errors .  Some  types  of  behavior  occur 
usually  under  conditions  that  are  very  difficult  to  observe, 
for  example  covert  or  illicit  acts.  Finally,  unless  prior 
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investigations  are  conducted,  it  is  very  difficult  to 
position  observers  in  settings  where  critical  behavior 
occurs  at  a  high  frequencies . 

Contrived  observation  substitutes  the  human  record¬ 
keeping  observers  present  in  simple  observation  with 
hardware  devices  (Webb  et  al . ,  1966).  Hardware  devices  are 
substituted  for  humans  as  means  of  avoiding  human  instrument 
error  and  supplementing  control  of  the  experiment.  It  is  an 
unobtrusive  form  of  data  collection,  employing  devices  such 
as  hidden  video  caimeras,  mirrors,  and  audio  recorders. 
Measurements  obtained  through  the  use  of  contrived 
observations  share  the  saune  sources  of  threats  to  validity 
as  measurements  obtained  through  the  use  of  simple 
observation . 

Participant  observation  of  behavior  is  a  field 
strategy  that  simultaneously  combines  document  amalysis, 
respondent  and  informant  interviewing,  direct  participation 
and  observation,  and  introspection  (Denzin,  1970) .  The 
observer  can  ass\ime  one  of  four  roles. 

1.  Complete  participant  role  is  one  in  which  the  observer 
is  wholly  concealed  and  he  tries  to  become  a  full 
member  of  the  investigated  group. 

2.  Participant  as  observer  role  is  when  the  investigated 
group  is  aware  of  the  identity  and  intentions  of  the 
observer . 

3.  Observer  as  participant  role  is  one  in  which 
investigations  usually  involve  only  one  visit  or 
interview  with  the  respondent.  The  interaction  is 
typically  brief  and  highly  formalized,  using 
questionnaires  or  interviews .  No  attempt  is  made  by 
the  observer  to  establish  a  sense  of  a  lasting 
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relationship  with  the  respondent--the  epitome  of  the 
stranger  encounter. 

4 .  The  complete  observer  role  involves  complete  removal 
of  the  observer  from  the  interaction.  The  complete 
observer  role  is  an  unobtrusive  method  of  collecting 
data,  while  the  other  roles  are  obtrusive. 

Denzin  (1970)  identifies  factors  regarding 
participant  observation  that  pose  threats  to  internal 
validity . 

1 .  Historical  factors  are  events  that  either  occurred 
before  observations  were  made  or  during  the  period 
between  the  first  and  last  observations. 

2 .  Subject  maturation  factors  reflect  the  changes  that 
occur  in  respondents,  resulting  from  their 
relationships  with  observers. 

3.  Subject  bias  factors  are  the  nonhomogeneous 
characteristics  of  the  observed  subjects. 

4 .  Subject  mortality  factors  reflect  the  nvunber  of 
subjects  who  drop-out  of  or  are  dropped  from  the 
experiment . 

5.  Reactive  effects  of  the  observer  factors  are  effects 
that  are  created  through  the  introduction  of  the 
observer . 

6.  Observer  changes  factors  are  effects  that  are 
attributed  to  changes  occurring  in  observers . 

7.  Situation  factors  are  effects  caused  by  peculiar 
aspects  of  the  situation  in  which  the  observations 
were  conducted. 

The  degree  to  which  the  saunple  selected  for  observation 
actually  represents  larger  populations  to  which  study 
findings  are  generalized  poses  concerns  of  external 
validity . 
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2 .  Experiments 

Ceimpbell  and  Stanley  (1966)  define  experiments  as 

that  portion  of  research  in  which  varizdDles  are  manipulated 

in  controlled  settings  and  their  effects  upon  other 

variables  observed.  Denzin  (1970)  states, 

"In  its  generic  form  the  experiment  represents  a 
situation  in  which  the  investigator  controls  some 
variables  while  he  manipulates  the  effects  on  others . 

This  permits  him  to  observe  the  effects  of  the 
manipulated  variables  upon  the  dependent  varisible  in  a 
situation  where  the  effect  of  other  relevant  factors  is 
believed  to  have  been  removed,  typically  by 
randomization . " 

Experiments  are  an  obtrusive  method  of  data  collection. 
Figure  2  represents  the  characteristics  of  a  generic 
experiment . 
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Figure  2.  Pr«t«st  and  Post-test  One  Control  Group  Design 


As  discussed  in  Chapter  II,  Sections  H  and  I, 
Campbell  and  Stanley  (1966)  identified  12  factors 
jeopardizing  the  internal  or  external  validity  of 
experimental  designs.  Denzin  (1970)  argues  that  the 
experiment  is  the  model  for  causal  inference  from  which  all 
other  inferential  strategies  are  either  derived  or  to  which 
they  are  compared.  Despite  the  formidable  number  of  threats 


18 


to  the  validity  of  experiments,  they  provide  one  of  the  best 
methods  to  answer  questions  of  causal  inference .  The  use  of 
experimental  methods  conveys  scientific  meaning  to  a  study's 
findings . 

3 .  Reports  by  Others 

Researchers  often  use  informants  to  secure  reports 
from  others.  LeVine  (1981)  expands  on  previous  works  of 
Campbell  to  define  an  informant  as  normally  a  member  of  a 
group  who  draws  and  communicates  generalizations  from  the 
particularities  of  his  social  environment  to  a  researcher. 
The  informant  generally  possesses  superior  inductive  or 
verbal  skills  or  has  experienced  exposure  to  the  phenomena 
in  which  the  researcher  is  interested.  Denzin  (1970) 
further  defines  the  role  of  the  informant  as  an  unobtrusive 
de  facto  observer  for  the  researcher,  an  intimate,  a 
confidant,  and  a  friend.  The  presence  of  this  kind  of 
binding  relationship  between  the  informant  and  researcher 
distinguishes  an  informant  from  a  respondent.  LeVine  (1981) 
identifies  the  informant's  biases  as  this  method's  primary 
threat  to  validity.  Informants  view  their  environment  from 
a  particular  vantage  point.  This  leads  inevitaJDly  to  biased 
perspective,  which  poises  threats  to  the  internal  and 
external  validity  of  findings  derived  from  measurements 
obtained  through  informants . 
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4 .  Analyses  of  Existing  Records  and  Documents 

Webb  et  al.,  (1966)  state  analyses  of  existing 
records  and  docximents  are  an  unobtrusive  method  of 
collecting  data.  Researchers  access  archives  containing 
either  vocational  or  scholarly  docviments  and  records  to 
obtain  relevauit  data.  Computer  usage  files  are  an  example 
of  archival  records . 

The  selective  deposit  and  selective  survival  of 
information  in  archives  can  introduce  a  significant  degree 
of  bias,  thereby  threatening  both  internal  and  external 
validity.  Selective  survival  occurs  when  existing  records 
and  documents  are  not  a  representative  ssonpling  of  the 
studied  population.  Selective  deposit  occurs  when  records 
and  docximents  are  generated  on  a  non-random  basis . 

5.  Interviews  and  Questioxmaires 

An  interview  is  a  form  of  survey  study  that  involves 
"a  face  to  face  verbal  interchange  in  which  one  person,  the 
interviewer,  attempts  to  elicit  information  or  expressions 
of  opinions  or  belief  from  another  person  (Maccoby  and 
Maccoby,  1954)."  Interviews  are  an  obtrusive  method  of  data 
collection.  The  basis  of  all  interviews  is  the  questions 
that  are  asked.  Denzin  (1970)  comments  on  interviews,  "The 
questionnaire  or  interview  must  serve  two  broad  purposes: 
it  must  translate  research  objectives  into  specific 
questions,  the  answers  to  which  will  provide  data  necessary 
for  hypothesis  testing,  and  it  must  assist  the  interviewer 
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in  motivating  the  respondent  so  that  the  necessary 
information  is  given."  An  interview  is  conducted  in  one  of 
three  forms: 

1 .  Schedule  standardized 

2 .  Nonschedule  standardized 

3.  Nonstandardized. 

The  schedule  standardized  interview  involves  wording 
and  ordering  all  questions  exactly  the  Scune  for  all 
respondents,  thus  using  a  uniform  measuring  instrument.  It 
must  be  assximed  that  the  questions  meaui  the  Scune  thing  to 
all  respondents .  Granting  that  all  questions  are 
comparcible,  any  variations  between  respondents  are 
attributed  to  actual  differences  in  the  response,  not  the 
measuring  instrument  (Denzin,  1970) . 

The  nonschedule  standardized  interview  involves  the 
interviewer  working  with  a  standard  list  of  the  information 
required  from  each  respondent,  but  ordering  emd  phrasing  the 
questions  in  such  a  manner  as  to  best  gather  the 
information.  This  allows  the  interviewer  to  partially 
tailor  the  interview  to  the  individual  characteristics  of 
each  respondent .  Modification  of  the  interview  requires  a 
highly  skilled  interviewer.  Granting  that  the  interviewer 
succeeds  in  asking  comparadsle  questions  to  all  respondents 
during  such  nonschedule  standardized  interviews,  any 
differences  are  attributed  to  actual  variances  in  response, 
not  the  measuring  instrument  (Denzin,  1970) . 


The  nonstandardized  interview  does  not  use  a  pre¬ 
specified  set  of  questions  nor  are  questions  asked  in  a 
specific  order.  This  allows  the  interviewer  complete 
freedom  to  explore  any  issues  that  arise  during  the  course 
of  the  interview.  It  is  clearly  recognized  that  respondents 
are  not  presented  with  a  comparable  set  of  questions 
(Denzin,  1970) . 

A  questionnaire  is  another  form  of  survey  study  that 
is  quite  similar  to  a  scheduled  standardized  interview. 
Questionnaires,  like  an  interview,  attempt  to  elicit 
information  or  expressions  of  opinions  or  belief  from 
another  person  or  persons.  They  differ  from  the  interview 
in  that  the  interchange  is  not  verbal .  The  interchange  most 
commonly  occurs  in  the  form  of  written  questions  and 
answers.  Questionnaires  are  an  obtrusive  method  of  data 
collection . 

Denzin  (1970)  declares  that  all  types  of  interviews 
and  questionnaires  share  common  deficiencies.  First, 
researchers  relying  on  interviews  and  questionnaires  can 
experience  difficulties  penetrating  the  respondent's  group 
language  and  mechanisms  of  symbolism.  When  failure  is 
experienced,  they  often  ask  questions  concerning  what  is  on 
the  mind  of  the  researcher,  rather  than  what  is  on  the  mind 
of  the  rean indent . 

Second,  errors  arise  from  respondents  not  always 
interchanging  honestly  with  the  interviewer  during  the 
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conduct  of  an  interview  or  completion  of  a  questionnaire.  A 
variation  of  this  deficiency  can  occur  when  questionnaires 
are  mailed  to  targeted  respondents .  Persons  other  than  the 
targeted  respondent  can  complete  mailed  questionnaires, 
assxuning  the  identity  of  targeted  respondents .  Researchers 
typically  assume  that  questionnaires  are  completed  by  the 
targeted  respondent,  since  they  possess  little  means  of 
knowing  who  actually  completes  questionnaires. 

Third,  respondents  occupy  different  positions  within 
their  own  groups,  and  hence  have  their  own  interpretations 
and  even  distortions  of  what  are  the  actual  group's  values. 
Each  of  these  commonly  shared  deficiencies  potentially 
threaten  the  internal  and  external  validity  of  measurements 
when  using  the  interview  as  a  data  collection  technique. 

€ .  Physical  Tracas 

Webb,  et .  al .  (1966)  discussed  an  unobtrusive  data 

collection  technique  that  is  based  on  the  study  of  physical 
traces.  Researchers  analyze  physical  traces  and  signs  that 
were  left  behind  by  a  population.  The  producers  of  the 
physical  traces  and  signs  are  unaware  of  their  future  use  by 
researchers .  There  are  two  basic  types  of  physical  traces : 
erosion  measures  and  accretion  measures . 

Erosion  measures  are  remains  of  some  population' s 
activities  that  have  selectively  worn  certain  objects.  The 
path  worn  on  a  tile  floor  is  an  example  of  an  erosion 
measure.  Erosion  measures  are  susceptible  to  biases 
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resulting  from  the  physical  composition  of  materials  in 
which  the  traces  are  recorded. 

Accretion  measures  are  the  recording  of  population 
deposits  over  time.  The  cigarette  butts  deposited  in  an  ash 
tray  are  an  exaunple  of  an  accretion  measure.  Accretion 
measures  are  susceptible  to  biases  resulting  from  the 
possibility  that  materials  have  selectively  survived  or  been 
selectively  deposited.  Selective  survival  occurs  when  only 
a  non-representative  sampling  of  the  population  exists. 
Selective  deposit  occurs  when  physical  traces  are  created  on 
a  non-random  basis. 
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III.  METHODOLOGY 


A.  SELECTING  STUDIES  FOR  REVIEW 

1 .  Choosing  Sources  from  which  to  Obtain  Studies 

Journals  in  information  systems  and  related  fields, 
published  in  English  between  1970  and  1990,  were  the  source 
from  which  studies  of  information  systems  effectiveness  were 
selected  for  review.  These  journals  are  accepted  as 
appropriate  professional  reading  material  and  pxablishing 
outlets  for  information  systems  academics  (Davis,  1900; 
Hamilton  and  Ives,  1983;  Vogel  and  Wetherbe,  1984;  Alavi  et 
al . ,  1989).  The  journals  surveyed  were: 

1 .  Academy  of  Management  Journal 

2.  Accounting  Review 

3.  ACM  Computing  Surveys 

4 .  Administrative  Science  Quarterly 

5 .  Communications  of  the  ACM 

6.  Data  Base 

7 .  Datamation 

8 .  Decision  Sciences 

9 .  EDP  Analyzer 

10.  Harvard  Business  Review 

11.  IEEE  Computer 

12.  Information  and  Management 

13.  Interfaces 
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14. 


Journal  of  Management  Information  Systems 

15.  Journal  of  Systems  Management 

16.  Management  Information  Systems  Quarterly 

17 .  Management  Science 

1 8 .  Omega 

19.  Sloan  Management  Review 

2.  Identifying  Specific  Studies  for  Review 
a.  On-Line  Bibliographic  Search 

An  on-line  search  in  January,  1989  reported  128 
items  using  the  string  of  key  words:  (Productivity  or 
Effectiveness)  and  (Measures  or  Measurement  or  Assess  or 
Assessment)  and  Information  Systems.  Out  of  these,  18 
articles  appeared,  on  the  face  of  it,  to  report  empirical 
assessments  of  information  systems  effectiveness  as 
determined  by  a  reading  of  their  titles.  None  of  them, 
despite  having  been  published  in  information  systems 
journals  (e.g.  MIS  Quarterly^  Journal  of  MIS,  Information 
and  Management,  Data  Base)  ,  are  eunong  the  37  empirical 
studies  of  information  systems  effectiveness  discussed  here. 
Some  focused  on  the  development  of  survey  measurement 
instruments,  a  category  that  was,  as  it  is  explained  below, 
excluded  from  this  analysis.  Other  articles  were  from  trade 
and  industry  publications  aimed  at  MIS  practitioners  and 
dealt  with  advice  on  effectiveness  measurement.  They  did 
not  report  empirical  studies  of  system  effectiveness. 
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b.  Manual  Bibliographic  Search 

It  was  suspected  that  the  empirical  literature 
on  information  systems  effectiveness  was  more  extensive  than 
the  handful  of  references  revealed  by  the  on-line  search.  A 
manual  search  of  the  Scune  journals  was  conducted.  Every 
issue  of  all  the  journals,  dating  back  to  1970,  were 
searched.  The  manual  searching  brought  forth  the  difficulty 
with  the  on-line  search;  deficient  key  words.  Authors,  for 
example,  used  key  words  such  as  "manufacturing  and 
engineering  systems,  return  on  investment,  utilization 
process,  human  information  processing,  information  system 
design,  system  evaluation,  information  system  investment, 
cost /benefit  calculations,  management  reporting,  management 
issues,  accuracy,  content,  information  characteristics  and 
users  evaluation."  Such  key  words  were,  in  retrospect, 
quite  appropriate  for  describing  various  aspects  evaluating 
information  systems  effectiveness.  They  simply  were  not  the 
ones  that  had  been  imagined  to  be  relevant  to  the  on-line 
search.  No  single  search  could  have  accommodated  so  many 
key  words .  While  a  sequence  of  searches  could  have  employed 
a  large  number  of  key  words,  the  limited  on-line  searching 
resources  available  for  this  study  did  not  make  that 
alternative  feasible. 

The  manual  effort  produced  118  articles  that 
were  identified  (by  title,  key  words  and  abstracts)  as 
dealing  with  the  evaluation  of  information  systems 


27 


effectiveness.  The  reference  lists  in  these  articles,  in 
turn,  identified  another  31  articles  that  had  been 
overlooked  because  of  title,  bringing  the  total  niainber  of 
items  in  the  literature  base  to  149.  Most  of  these  articles 
did  not  report  empirical  studies . 

c.  Selection  of  Specific  Studies 

Excluded  immediately  were  non-empirical  analyses 
of  the  assessment  of  information  systems  (Chervany  and 
Dickson,  1970;  Coleman  and  Riley,  1972;  Mason  and  Mitroff, 
1973;  Carlson,  1974;  Soergel,  1976;  Strassmann,  1976;  Tharp 
and  Taggart,  1976;  Radecki,  1976;  Dickson  et  al . ,  1977; 
Arnovick  and  Gee,  1978;  King  and  Schrems,  1978;  Ginrberg, 
1978;  Mason,  1978;  Kleijnen,  1979;  Worthley,  1980;  Ginzberg, 
1980;  Hamilton  and  Ives,  1982;  Borko,  1983;  Keim  and  Janaro, 
1982;  Markus  and  Robey,  1983;  Mason,  1984;  Calista,  1985; 
Bearman  et  al.,  1985;  Gremillion  and  Pyburn,  1985;  Lay, 

1985;  Maggiolini,  1986;  Martin  euid  Trumbley,  1986;  Rouse, 
1987;  Marsden  aind  Pingry,  1988;  Trice  and  Treacy,  1988)  . 

Also  excluded  from  this  review  were  empirically- 
based  articles  that  do  not  deal  directly  with  a  study  of 
information  systems  effectiveness  but  rather  were  concerned 
with  the  validation  of  attitudinal  questionnaires  and  other 
instruments  (Larcker  and  Lessig,  1980;  Mick  et  al . ,  1980; 
Chandler,  1982;  Bailey  and  Pearson,  1983;  Srinivasan,  1985; 
Dominick,  1987;  Hill  et  al . ,  1987;  Miller  and  Doyle,  1987; 
Baroudi  and  Orlikowski,  1988;  Money  et  al . ,  1988;  Montezema, 
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1988) .  While  these  works  involve  the  development  of 
research  techniques  that  contribute  to  how  we  know  what  we 
know  about  information  systems  effectiveness,  they  were  left 
out  because  they  were  not  primarily  engaged  in  studies  that 
had  a  potential  for  developing  theory  aJ^out  behavior  related 
to  information  systems  effectiveness. 

A  miscellany  of  pieces  that  seemed  to  have  the 
titles  sought  but  actually  dealt  with  other  issues  were  also 
removed  from  the  original  149  articles.  Those  addressing 
evaluations  of  the  technical  performance  of  hardware  (e.g. 
computer  processor  performance  in  millions  of  instructions 
per  second)  were  not  included.  Similarly,  articles  dealing 
with  efficiency  in  software  development  activities  (e.g. 
programmer  productivity  in  terms  of  lines  of  code  written 
per  day)  were  excluded. 

Finally,  my  search  efforts  boiled  down  to  the  37 
reports  of  empirical  studies  of  the  effectiveness  of 
information  systems  in  its  various  formulations.  A  listing 
of  these  reports  is  provided  at  Appendix  A.  Surely,  for  all 
of  this,  some  studies  may  have  been  missed,  possibly  some 
important  studies.  For  that  I  apologize  in  advance  to  those 
researchers  who  might  have  been  overlooked.  It  is  my 
contention  that  even  though  every  last  empirical  study  of 
information  systems  effectiveness  may  not  have  been  included 
in  this  research,  I  do  believe  that  those  included  are 
representative  of  the  field. 
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B.  REVIEWING  THE  STUDIES 


The  37  empirical  studies  (listed  in  Appendix  A)  were 
individually  reviewed  to  identify  data  collection 
instruments  and  discussions  of  validity  issues.  Researchers 
often  used  multiple  data  collection  instruments  in  the 
conduct  of  their  studies,  with  each  instrument  contributing 
only  a  portion  of  the  measurements  upon  which  the 
researchers'  findings  were  based.  This  use  of  multiple  data 
collection  instruments  necessitated  the  development  of  a 
methodology  to  identify  and  separately  analyze  each 
instrument  used  in  the  37  studies . 

Each  study  was  carefully  read  to  identify  all  data 
collection  instruments  contributing  to  the  findings  of  the 
study.  Each  instrument  was  further  analyzed  to  record 
information  concerning  its  design,  purported  focus  of 
measurement,  administration,  and  validity.  The  analysis 
process  for  each  study  essentially  consisted  of  the 
following; 

1 .  Identification  and  categorization  of  all  data 
collection  instruments  used  by  the  information  systems 
researchers  for  the  given  study. 

2 .  Categorization  of  each  identified  data  collection 
instrxament  according  to  what  the  information  systems 
researchers  purported  it  to  measure . 

3.  Excunination  of  the  way  in  which  each  identified  data 
collection  instrument  were  administered. 

4.  Identification  of  how  the  information  systems 
researchers  dealt  with  the  validity  of  each  identified 
data  collection  instrument . 
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The  primary  manner  in  which  data  we:  2  collected  during 
the  analysis  of  each  data  collection  instrument  was  through 
the  careful  reading  of  the  37  studies .  Pertinent 
information  was  identified  and  extracted  from  the  studies, 
clearly  requiring  the  exercise  of  personal  judgment . 
Consequently,  all  findings  resulting  from  this  research  are 
ultimately  based  upon  face  validity.  The  remainder  of  this 
section  provides  details  on  how  the  analysis  was  conducted 
and  recorded. 

1 .  Identification  &.iid  Categorization  of  Instruments 

The  initial  goal  of  the  review  was  to  identify  data 
collection  instruments  used  by  the  information  systems 
researchers  to  measure  aspects  of  information  systems . 
Instrviments  that  were  used  solely  in  the  design  of  other 
measuring  instruments,  for  example  preliminary  interviews 
conducted  to  design  questions  appearing  later  in  a 
questionnaire,  were  not  considered  of  interest  to  this 
research . 

Each  data  collection  instrument  was  classified  by 
the  data  collection  technique  it  primarily  employed. 
Instruments  were  classified  as  cpjestionnaires,  interviews, 
archival  records,  and  observations. 

Each  data  collection  instrument  was  further  analyzed 
to  determine  specific  design  characteristics  of  the 
technique.  Data  were  gathered  on  the  specific  design  of 
each  questionnaire,  interview,  collection  of  archival 
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records,  or  set  of  observations  used  by  the  researchers. 

The  type  of  scaling  and  original  source  of  each 
questionnaires  was  recorded.  The  amount  of  schedule  and 
structure  existing  in  each  interview  was  recorded.  The 
source  and  types  of  records  used  for  each  collection  of 
archival  data  was  recorded.  The  manner  in  which 
observations  were  conducted  was  recorded. 

2 .  Categorization  of  Instruments  by  Measures 

It  is  of  interest  to  determine  what  information 
systems  researchers  purported  to  measure  with  data 
collection  instr\aments .  Each  instrument  is  purported  by  its 
designer  to  provide  one  or  more  general  types  of 
measurements  relating  to  information  systems .  This  study 
categorizes  instruments  according  to  the  type  or  types  of 
measurements  it  purportedly  records.  The  instruments  used 
by  researchers  were  categorized  as  purportedly  measures  of 
the  following: 

1 .  Attitude  or  perception 

2 .  Behavior 

3 .  System  characteristics 

4.  Situational  characteristics. 

Each  instrument  used  in  the  37  studies  was  analyzed  to 
determine  the  category  or  categories  of  measures  that  its 
designers  claimed  it  provided. 

Measures  of  attitude  or  perception  are  provided  by 
respondents  when  solicited  by  researchers.  This  usually 
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involves  respondents  providing  opinions  or  testimony  on  some 
aspect  of  an  information  system.  Consequently,  the 
researchers  are  indirectly  measuring  the  information  system 
through  the  observations  provided  by  others .  Measures  of 
attitude  or  perception  typically  involve  aspects  of 
information  systems  such  as  user  involvement,  effectiveness 
of  systems,  and  satisfaction  with  systems. 

Measures  of  behavior  record  the  occurrence  of 
behavior  related  to  information  systems.  The  behavior 
usually  takes  the  form  of  interactions  between  users  and 
information  systems  or  actions  taken  as  a  result  of  outputs 
provided  by  information  systems.  Measures  of  behavior 
usually  involve  the  researchers  directly  observing  or 
deducing  specific  behavior  relative  to  information  systems 
or  their  users ,  Observation  and  archival  records  are  often 
used  to  collect  measures  of  behavior.  Data  recorded  through 
the  review  of  computer  inquiry  logs  or  account  records 
regarding  system  usage  is  an  example  of  measures  of 
behavior . 

Measures  of  system  characteristics  are  data  which 
purport  to  directly  represent  some  aspects  of  information 
systems.  Examples  are  cost  of  systems,  the  effectiveness  of 
systems,  the  impact  of  systems,  the  success  of  systems,  or 
the  satisfaction  of  users .  Measures  of  system 
characteristics  differ  from  measure  of  attitude  or  behavior 
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in  that  they  are  purported  to  directly  measure  aspects  of 
information  systems . 

Measures  of  situational  characteristics  provide  data 
concerning  the  environment  within  which  the  system  operates . 
Measures  of  situational  characteristics  are  concerned  with 
describing  components  of  the  environment  that  essentially 
impact  upon  information  systems.  Measures  of  situational 
characteristics  differ  from  the  previously  discussed  types 
of  measures  in  that  they  do  not  record  data  on  the  impact  of 
the  studied  information  systems;  however  they  do  provide 
insight  into  the  conditions  surrounding  information  systems. 
Examples  of  components  of  the  environment  that  are  measured 
by  researchers  are  organizational  structure,  historical 
performance  of  business,  personnel  profiles,  and  industry 
trends . 

3.  Examination  of  tha  Way  Inatrumants  ara  Administarad 

The  way  in  which  data  collection  instruments  are 
administered  is  an  essential  part  of  the  design  of  an 
experiment .  It  involves  the  selection  of  potential 
respondents,  distribution  of  the  data  collection 
instruments,  and  follow-up  actions  taken  to  encourage 
response  and  account  for  non-response.  The  manner  in  which 
researchers  administer  instruments  can  fundamentally  affect 
the  validity  of  data. 

The  goal  of  this  examination  is  to  provide 
information  on  how  researchers  are  administering  data 
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collection  instruments.  Analysis  of  the  sampling  process 
used  by  researchers  may  provide  information  regarding  the 
validity  of  the  data  they  collect.  Each  identified 
instrument  was  examined  to  determine  the  following: 

1 .  Who  responded  to  each  instrument . 

2.  The  size  of  the  Scimple  taken  with  each 
instrument . 

3.  The  response  rate  achieved  for  each  instrument,  with 
the  exception  of  archival  records. 

4.  How  each  instrximent  was  distributed. 

5.  Whether  follow-up  actions  were  taken  to  improve  the 
participant  response  rate  for  each  instrument. 

6.  Whether  differences  between  respondents  and  non¬ 
respondents  to  each  instrxament  were  identified  by  the 
information  systems  researchers . 

4 .  Identification  of  how  Researchers  dealt  with 
Validity  Issues 

The  primary  focus  of  this  study  is  to  explore 
validity  issues  stemming  from  measurements  of  information 
systems.  Identifying  how  information  systems  researchers 
discussed,  claimed,  and  substantiated  the  validity  of  data 
collected  during  their  studies  provides  a  basis  for 
assessing  the  role  validity  issues  play  in  information 
systems  research.  The  following  information  was  recorded 
for  each  data  collection  instrxjment  in  order  to  establish 
how  researchers  dealt  with  validity  issues: 

1.  How  internal  validity  was  addressed. 

2.  How  external  validity  was  addressed. 
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3.  The  types  of  validity  claimed  or  implied. 

4.  The  source  of  the  validity  claimed  or  implied. 

All  discussion  regarding  the  validity  of 
measurements  derived  from  a  given  instruments  were  recorded. 
Secondary  sources  were  examined  and  noted  when  cited  in  a 
discussions .  All  recorded  discussions  were  then  determined 
to  address  internal  and  or  external  validity  issues,  and 
classified  as  such.  The  classified  discussions  were  then 
further  summarized  for  ease  of  understanding.  When  no 
discussions  of  internal  and  or  external  validity  were 
presented,  it  was  so  noted  as  not  discussed.  Further 
analysis  was  not  required  for  instruments  without 
discussions  of  internal  and  external  validity. 

Data  collection  instruments  having  discussions  of 
internal  and  or  external  validity  were  further  analyzed. 

The  discussions  were  determined  to  claim,  imply,  or  deny  the 
validity  of  measurements.  The  types  of  validity  claimed  or 
implied  were  noted,  as  well  as  denials  of  validity.  Whether 
validity  was  claimed  or  implied  was  also  noted. 

When  the  validity  of  measurements  was  claimed  or 
implied  for  a  given  data  collection  instrviment,  the  source 
of  the  validity  was  noted.  It  was  also  noted  when 
information  systems  researchers  claimed  or  implied  validity 
but  failed  to  cite  a  source. 

Discussions  that  collectively  address  the  validity 
of  measurements  derived  using  multiple  data  collection 
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instruments  in  a  given  study  were  recorded.  These 
discussions  were  further  analyzed  as  if  they  represented  a 
single  data  collection  instrument.  The  results  of  such 
analyses  are  identified  as  "collectively",  when  such 
discussion  were  present  in  a  study. 

C.  RECORDING  THE  DATA 

Data  were  extracted  from  the  37  studies  using  a  separate 
svimmary  sheet  for  each  study.  (Appendix  B)  In  order  to 
consolidate  the  siommary  sheets  into  a  more  understandaJDle 
format,  a  matrix  was  created  to  present  the  data  as  detailed 
in  the  previous  section.  (Appendix  C)  Both  the  svimmary 
sheets  and  the  matrix  are  used  as  sources  for  the  findings 
of  this  study. 
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IV.  FINDINGS 


Appendix  B  contains  a  detailed  analysis  of  the  37 
analyzed  studies.  Appendix  C  is  a  consolidation  of  data 
presented  in  Appendix  B.  Both  of  these  appendices  were  used 
as  sources  for  the  findings  that  follow. 

A.  INSTRUMENTS  USED  BY  RESSIARCHERS 
1 .  Summary  of  Instruments 

The  number  and  types  of  data  collection  instriiments 
used  by  information  systems  researchers  during  the  37 
studies  are  shown  in  Table  1 . 

TABLE  1 

DATA  COLLECTION  INSTRUMENTS  USED 


INSTRUMENT 

#  STUDIES  USING 

#  TIMES  USED 

Questionnaire 

29 

36 

Archival 

14 

15 

Interview 

8 

9 

Observation 

4 

4 

Unidentified 

1 

1 

The  column  labeled  #  STUDIES  USING  contains  data 
regarding  the  number  of  studies  that  used  specified  types  of 
instruments .  The  column  labeled  #  TIMES  USED  contains  data 
regarding  the  number  of  times  each  type  of  instrument  was 
used  during  the  37  studies.  The  number  of  instruments  used 
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exceeded  the  number  of  studies  because  some  researchers  used 


multiple  data  collection  instruments  during  the  conduct  of  a 
single  study. 

Table  1  reveals  that  questionnaires  were  used  in 
over  three-quarters  of  the  studies.  Twenty-nine  of  37 
studies  included  questionnaires.  Data  collection  which 
involved  archival  records,  interviews,  or  observations  was 
less  frequent.  When  these  latter  data  collection  techniques 
were  used,  they  were  often  used  in  conjunction  with 
questionnaires . 

The  37  studies  used  65  individual  data  collection 
techniques.  Over  half  of  the  data  collection  techniques 
used  were  questionnaires,  with  36  of  the  65  techniques 
having  been  classified  as  questionnaires. 

2 .  Use  of  Multiple  Data  Collection  Instzruments 

Table  2  illustrates  the  use  of  multiple  data 
collection  instruments  by  information  systems  researchers. 
Use  of  multiple  data  collection  instruments  denotes  the 
incorporation  of  more  than  one  data  collection  instrument  in 
a  given  study.  For  exeunple,  Lucas  (1974)  used  two 
questionnaires  and  Colton  (1972)  used  a  questionnaire  and  an 
interview,  thus  each  used  multiple  data  collection 
instruments.  Table  2  shows  the  number  of  studies  that  used 
one,  two,  three,  or  four  data  collection  techniques. 
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TABLE  2 


USE  OF  MULTIPLE  DATA  COLLECTION  INSTRUMENTS 


NUMBEI 

One 

1  OF  ins: 

Two 

rRUMENTS 

Three 

USED 

Four 

NUMBER  OF  STUDIES 

19 

9 

8 

1 

TaJDle  2  shows  that  over  half  of  the  researchers  used 
only  one  instrument  in  their  research  designs .  A  further 
analysis  of  the  19  one-technique  studies  revealed  that  they 
used  ten  questionnaires,  six  sets  of  archival  records,  two 
collections  of  observations,  and  one  interview. 

3 .  Characteristics  of  Questionnaires 

Details  concerning  design  characteristics  were 
provided  for  31  of  the  36  questionnaires  used  in  the  37 
studies .  Even  when  discussions  were  presented,  they  rarely 
extended  beyond  the  identification  of  the  types  of  scales 
used  to  score  instruments.  Few  studies  clearly  identified 
the  origin  or  source  of  questionnaires  used.  Discussions 
were  insufficient  to  develop  findings  ad^out  the  structure  of 
questionnaires . 

Questionnaires  used  by  researchers  were  largely 
based  upon  ordinal  scales.  Of  the  31  questionnaires  that 
discussed  their  characteristics,  30  were  based  upon  ordinal 
scales.  Five,  seven,  and  ten  point  ordinal  scales  were 
typically  used  in  the  design  of  the  questionnaires.  Five 
studies  contained  discussions  specifically  stating  that 
their  corresponding  questionnaires  were  Likert-type;  Franz 
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and  Robey  (1986),  Aldag  and  Power  (1986),  Franz  et  al . 

(1986) ,  Gallupe  and  DeSanctis  (1988) ,  Tait  and  Vessey 
(1988)  . 

The  origin  of  the  questionnaires  were  discussed  for 
a  third  of  the  questionnaires  used.  The  aJosence  of 
discussions  left  uncertain  whether  the  questionnaires  were 
of  original  design  or  were  developed  during  previous 
studies.  While  none  of  the  questionnaires  were  explicitly 
identified  as  having  been  of  original  design,  however,  it  is 
possible  that  some  were.  Eleven  of  the  36  questionnaires 
were  identified  as  developed  during  previous  studies.  These 
questionnaires  were  attributed  to  one  or  more  of  the 
following;  Smith  et  al.  (1969),  Lucas  (1973),  Schultz  and 
Slevin  (1975),  Wolek  (1975),  Elizur  and  Guttman  (1976), 

Sims  et  al.  (1976),  Ein-Dor  and  Segev  (1978),  Keen  (1981), 
McFarlan  (1981),  Bailey  and  Pearson  (1983),  Ives  et  al . 

(1983) ,  Baroudi  and  Orlikowski  (1988) ,  and  Doll  and 
Torkzaheh  (1988) . 

4 .  Sources  of  Archival  Records 

Researchers  used  archival  records  15  times  in  the  37 
studies .  They  provided  discussions  identifying  the  sources 
of  the  archival  records  for  13  of  the  15  instances  in  which 
they  were  used.  Over  two-thirds,  9  of  the  13  discussions, 
identified  the  archival  records  as  computer  files  or 
computer  generated  reports.  Insufficient  information  was 
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provided  in  the  discussions  to  clearly  esteiblish  the  sources 
of  the  remaining  archival  records . 

5.  Characteristics  of  Interviews 

Nine  interviews  were  identified  in  the  37  studies. 
Little  discussion  was  presented  regarding  these  interviews. 
No  determination  of  structure  or  schedule  could  be  made  for 
six  of  the  nine  interviews.  Three  interviews  cited 
combinations  of  structured,  unstructured,  scheduled,  and 
unscheduled  designs . 

6.  Characteristics  of  Observations 

Observations  were  identified  four  times  as  the  data 
collection  technique  in  the  37  studies .  The  researchers 
discussed  the  manner  in  which  the  observations  were  made 
each  time  they  were  identified  as  used.  These  discussions 
were  used  to  determine  observer  roles,  as  defined  by  Denzin 
(1970) .  Three  of  the  observations  were  collected  by 
participemt  observers .  The  fourth  set  was  collected  by  a 
complete  observer . 

B.  NHAT  RESSARCBERS  PURPORTED  TO  MEASURE 

Instruments  used  in  the  37  studies  were  categorized  as 
purported  to  measure  of  attitude,  behavior,  system 
characteristics,  or  situational  characteristics.  Measures 
of  attitude  typically  consisted  of  stated  opinions  or 
testimony  as  provided  by  respondents .  Measures  of  behavior 
recorded  the  occurrence  of  behavior  related  to  information 
systems .  Measures  of  system  characteristics  are  data  which 
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are  purported  to  directly  represent  aspects  of  information 
systems .  Measures  of  situational  characteristics  provide 
data  concerning  the  environment  within  which  systems 
operate.  Further  explanations  of  these  categories  of 
measures  are  provided  in  Chapter  III. 

Table  3  shows  what  the  researchers  purported  to  measure 
in  the  37  studies.  Some  instrrunents  purportedly  collected 
more  than  one  category  of  measures.  For  exaunple,  Swanson 
(1974)  used  a  single  questionnaire  to  purportedly  measure 
both  attitude  and  behavior.  Consequently,  the  total  number 
of  purported  measures  is  greater  than  the  total  number  of 
instruments  used. 

TABLB  3 

PURPORTED  MEASURES  OF  INSTRUMENTS 


TYPE  of 

DATA  COLLECTION 
INSTRUMENT 

#  of 

TIMES 

USED 

by  ins: 
System 

NUMBER  C 

:rument  t: 

Attitude 

)f  PURPORl 
:PE  and  Ci 
Behavior 

[■ED  MEASURE 
VTEGORY  of 
Situation 

:s 

MEASURE 

Unknown 

Questionnaire 

36 

13 

25 

11 

6 

1 

Archival 

15 

5 

4 

6 

Interview 

9 

5 

2 

2 

Observation 

4 

4 

Unknown 

1 

1 

TOTAL 

65 

27 

27 

16 

14 

1 

Table  3  shows  that  the  65  instruments  made  a  combined 
total  of  85  purported  measures,  distributed  between  four 
categories.  Questionnaires  were  the  only  instrument 
identified  to  purportedly  measure  multiple  categories  of 
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measures  with  a  single  data  collection  instrviment . 
Questionnaires  were  also  the  only  instrument  identified  to 
purportedly  measure  all  categories  of  measures. 
Questionnaires  and  interviews,  when  considered  together, 
were  identified  27  times  as  purported  measures  of  attitudes 
and  38  times  as  purported  measures  of  the  remaining 
categories.  Analysis  of  archival  records  were  identified  to 
purportedly  measure  all  categories  except  attitudes. 
Observations  were  identified  solely  as  purported  measures  of 
systems  characteristics. 

C.  THE  NAY  IN  WHICH  INSTRUMENTS  WERE  ADMINISTERED 

1 .  Selection  of  Sample  Populations 

The  information  systems  researchers  szunpled 
populations  from  a  variety  of  professions  and  functional 
specialties .  All  37  studies  appear  to  have  used  samples 
selected  for  convenience  or  availad>ility .  Randomness  was 
introduced  into  the  sampling  process  only  after  the  sample 
populations  were  identified. 

2 .  Identity  of  Respondents 

A  total  of  49  questionnaires,  interviews,  and 
observations  were  used  in  the  37  studies.  Managers, 
computer  personnel,  and  computer  users  were  identified  as 
respondents  or  observed  subjects  for  43  of  the  49 
instruments .  The  remaining  six  instruments  used  students  as 
respondents  or  observed  subjects .  King  and  Rodriguez 
(1978),  Aldag  and  Power  (1986),  Durand  et  al .  (1987),  and 
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Gallupe  and  DeSanctis  (1988)  were  the  studies  that  used 
students.  All  other  studies  relied  on  respondents  or 
observed  subjects  that  used  information  systems  in 
operational  environments . 

3 .  Size  of  Samples 

Information  identifying  the  size,  duration,  or 
frequency  of  sampling  was  provided  for  only  38  of  the  65 
data  collection  instriiments .  When  such  information  was 
provided,  it  typically  consisted  of  identifying  only  the 
total  number  of  respondents  to  given  instruments.  The  total 
nximber  of  respondents  was  provided  for  34  instruments,  as 
shown  in  Te±ile  4 .  The  duration  of  sampling  was  provided  in 
three  studies;  Swanson  (1974) ,  Ball  (1980) ,  and  Ginzberg 
(1981) .  The  frequency  of  sampling  was  provided  in  one 
study,  Englander  and  Englander  (1985) . 

TABLE  4 
SAMPLE  SIZE 


1  to  50 

SAMPLE  SIZE 
51  to  100 

101  to  500 

#  of  INSTRUMENTS 

8 

12 

14 

4 .  Response  Rates 

A  total  of  40  questionnaires  and  interviews  were 
used  in  the  37  studies.  Researchers  identified  the  response 
rates  for  only  20  of  these  instruments.  Table  5  shows  the 
response  rates  that  were  identified  for  each  type  of 
instrument . 
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TABLE  5 


DISTRIBUTION  OF  IDENTIFIED  RESPONSE  RATES 


TYPE  OF 
INSTRUMENT 

0-20% 

RESPC 

21-40% 

)NSE  RATI 

41-60% 

— 

S  ACHIEVI 
61-80% 

SD 

81-100% 

Questionnaire 

1 

1 

 .....  .  . 

8 

8 

Interview 

1 

1 

TOTALS 

1 

1 

9 

9 

Further  analysis  of  the  response  rates  reveals  that 
the  researchers  rarely  reported  rates  higher  than  95 
percent.  Only  three  of  the  21  instruments  exceeded  a  95 
percent  response  rate;  questionnaires  used  by  Lucas  (1975b) , 
Robey  (1979) ,  and  Bruwer  (1984) .  A  100  percent  response 
rate  was  achieved  for  only  the  questionnaires  used  by  Robey 
(1979)  and  Bruwer  (1984) .  Unfortunately,  neither  study 
discussed  how  the  100  percent  response  rate  was  achieved. 

At  the  opposite  extreme,  the  findings  of  two  studies  were 
based  on  instruments  achieving  a  response  rate  of  less  than 
40  percent;  questionnaires  used  by  Snitkin  and  King  (1986) 
and  Baroudi  et  al.  (1986).  Snitkin  and  King  (1986)  achieved 
a  response  rate  of  18  percent  euid  Baroudi  et  al .  (1986)  of 

25  percent. 

5 .  Distribution  Methods 

Data  were  gathered  regarding  the  manner  in  which 
cjuest ionnaires  were  distributed  and  interviews  conducted.  A 
total  of  36  questionnaires  were  used  in  the  37  studies.  The 
means  of  distribution  was  clearly  identified  for  18  of  the 
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36  questionnaires.  Mail  was  identified  as  the  means  of 
distribution  for  15  of  the  questionnaires.  Three  of  the 
questionnaires  were  identified  as  administered  in  person. 

A  total  of  nine  interviews  were  used  in  the  37 
studies .  The  manner  in  which  the  interviews  were 
administered  was  identified  for  six  of  the  nine  interviews 
conducted.  These  six  interviews  were  identified  as 
administered  in  person.  No  interviews  appear  to  have  been 
administered  by  telephone . 

The  information  systems  researchers  provided  little 
discussion  regarding  actions  taken  to  improve  response 
rates.  A  total  of  40  instruments  used  questionnaires  and 
interviews,  yet  only  six  instances  of  follow-up  actions  were 
clearly  identified.  All  six  instances  involved 
questionnaires.  Colton  (1972)  used  a  second  mailing.  Lucas 
(1975a)  and  (1975b)  stated  that  unspecified  follow-up 
actions  were  taken  for  each  questionnaire.  Ginzberg  (1981) 
used  a  follow-up  message  for  his  two  questionnaires.  Tait 
and  Vessey  (1988)  used  a  telephone  call. 

6 .  Differences  between  Respondents  and  Non-Respondents 

The  40  data  collection  instruments  that  used 
questionnaires  or  interviews  were  exeunined  to  determine 
whether  researchers  discussed  any  differences  between 
respondents  and  non-respondents.  Discussion  of  differences 
were  presented  for  ten  of  these  instruments,  all  of  which 


47 


were  qnaestionnaires .  When  a  discussion  of  differences  was 
presented,  it  typically  provided  sparse  information. 

Ginzberg  (1981)  presented  data,  based  on  an  analysis 
of  both  groups,  suggesting  that  no  significant,  nor 
sizeaJDle,  difference  existed  between  respondents  and  non¬ 
respondents  to  his  three  questionnaires.  Cron  and  Sobol 
(1983)  specifically  identified  major  non-respondents  and 
discussed  differences.  Robey  (1979)  and  Bruwer  (1984) 
claimed  response  rates  of  100  percent,  thus  no  discussion  of 
non-respondents  was  necessary.  Snitkin  and  King  (1986) 
stated  they  were  prevented  from  performing  an  analysis  of 
differences.  Baroudi  et  ai.  (1986)  linked  non-respondents 
to  the  use  of  old  mailing  lists.  Franz  et  al .  (1986) 

performed  an  analysis  of  the  attitudes  of  dropouts  from  his 
experiment,  which  was  used  to  suggest  differences  between 
respondents  and  non-respondents  to  his  two  questionnaires . 

D.  HANDLING  OF  VALIDITY  ISSUES 

1.  Discuasiona  of  Internal  Validity 

Discussions  regarding  the  internal  validity  of 
individual  instr\iments  were  presented  for  only  19  of  the  65 
instruments.  Tadale  6  identifies,  by  type  of  instruments 
used,  the  presentation  of  such  discussions. 
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TABLE  6 


DISCUSSIONS  REGARDING  INTERNAL  VALIDITY 


TYPE  of 
INSTRUMENT 

TIMES 

USED 

TIMES 

DISCUSSED 

TIMES 

NOT  DISCUSSED 

Questionnaire 

36 

16 

20 

Archival 

15 

1 

14 

Interview 

9 

2 

7 

Observation 

4 

0 

4 

Unidentified 

1 

0 

1 

TOTALS 

65 

19 

46 

Analysis  of  the  19  discussions  reveals  that  the 
researchers  addressed  internal  validity  issues  in  a  variety 
of  ways.  References  detailing  ways  in  which  specific 
studies  addressed  internal  validity  are  provided  in 
subsequent  paragraphs .  Most  frequently  the  discussions 
identified  secondary  sources  as  the  original  developers  of 
their  instruments .  Some  discussions  clearly  cautioned 
readers  that  the  internal  validity  of  data  collection 
instruments  had  not  been  demonstrated.  Least  common  were 
discussions  that  asserted  that  internal  validity  had  been 
demonstrated . 

Eleven  of  the  discussions  cited  secondary  sources  as 
a  means  of  possibly  substantiating  the  internal  validity  of 
individual  data  collection  instruments;  Lucas  (1974),  Lucas 
(1975b),  King  and  Rodriguez  (1978),  Robey  and  Zeller  (1978), 
Lucas  (1978),  Robey  (1979),  Baroudi  et  al .  (1986),  Franz  et 
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ai .  (1986),  Tait  and  Vessey  (1988),  and  Cronan  and  Douglas 

(1990).  Appendix  B  identifies  the  specific  source (s)  cited 
by  each  of  these  studies.  The  citation  of  secondary  sources 
implies  that  the  individual  instruments  were  validated  in 
previous  studies.  One  can  expect  cited  secondary  sources  to 
provide  information  aibout  the  internal  validity  of 
individual  instruments. 

When  the  secondary  sources  cited  by  Lucas  (1974) , 
Lucas  (1975b) ,  King  and  Rodriguez  (1978) ,  Robey  and  Zeller 
(1978),  Robey  (1979),  and  Franz  et  al.  (1986)^  were 
examined,  little  discussion  of  internal  validity  was  found. 
The  cited  secondary  sources  that  provided  insufficient 
discussion  of  internal  validity  were  Lucas  (1973) ,  Schultz 
and  Slevin  (1975),  Wolek  (1975),  Ein-Dor  and  Segev  (1978), 
Keen  (1981) ,  and  McFarlan  (1981) . 

Secondary  sources  cited  by  Lucas  (1978) ,  Baroudi  et 
al .  (1986),  Franz  et  al .  (1986)*,  Tait  and  Vessey  (1988), 

and  Cronan  and  Douglas  (1990)  directly  discussed  the 
internal  validity  of  individual  instruments.  The  cited 
secondary  sources  that  provided  adequate  discussion  of 
internal  validity  were  Smith  et  al.  (1969),  Elizur  and 
Guttman  (1976),  Sims  et  al .  (1976),  Bailey  and  Pearson 


^Reference  to  secondary  source  of  Sims,  Szilagyi, 
Keller  (1976)  provided  discussion  of  validity  issues. 

^Reference  to  secondary  source  of  Schultz  and  Slevin 
(1975)  provided  little  discussion  of  validity  issues. 
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(1983),  Ives  et  al .  (1983),  Baroudi  and  Orlikowski  (1988), 

and  Doll  and  Torkzaheh  (1988) . 

Six  discussions  acknowledged  deficiencies  regarding 
the  internal  validity  of  individual  instruments.  Gallagher 
(1974)  informed  readers  that  the  validity  of  his 
questionnaire  had  not  been  established.  Edstrom  (1977) 
identified  weaknesses  in  his  interview  stemming  from 
problems  associated  with  human  memory,  perceptions,  and 
selection  of  respondents.  Englander  and  Englander  (1985) 
suggested  that  varied^les  not  measured  by  their  archival 
records  may  have  adversely  impacted  the  results  of  their 
experiment.  Lincoln  (1986)  addressed  the  unreliedDility  of 
user  interviews.  Baroudi  et  al.  (1986)  expressed  concerns 
over  the  ordering  of  experimental  varieJDles  and  the  use  of 
subjective  rather  than  objective  measures.  Tait  and  Vessey 
(1988)  acknowledged  problems  resulting  from  when  measures 
were  collected,  influences  on  respondent's  attitudes,  the 
use  of  Likert-type  scales,  and  insufficient  validation  of 
instruments . 

In  four  cases,  researchers  implied  or  asserted  that 
instruments  had  demonstrated  internal  validity.  Lucas 
(1978)  asserted  that  the  quasi-experimental  research  design 
and  statistical  techniques  used  provided  increased 
confidence.  Mansour  and  Watson  (1980)  implied  that  their 
questionnaire  possessed  internal  validity  on  the  basis  of 
its  relia±>ility  as  a  predictor.  Franz  and  Robey  (1986) 


51 


stated  that  they  developed  and  validated  their 
questionnaire.  They  provided  details  of  how  this  was 
accomplished.  Aldag  and  Power  (1986)  implied,  through 
persuasive  discussion,  that  their  questionnaire  had 
demonstrated  internal  validity. 

2 .  Discussions  of  External  Validity 

Information  systems  researchers  presented 
discussions  regarding  the  external  validity  of  individual 
instrtiments  for  only  22  of  the  65  instruments .  Table  7 
identifies,  by  type  of  instrument,  the  presentation  of  such 
discussions . 


TABLE  7 

DISCUSSIONS  REGARDING  EXTERNAL  VALIDITY 


TYPE  of 
INSTRUMENT 

TIMES 

USED 

TIMES 

DISCUSSED 

TIMES 

NOT  DISCUSSED 

Questionnaire 

36 

18 

18 

Archival 

15 

2 

13 

Interview 

9 

1 

8 

Observation 

4 

1 

3 

Unidentified 

1 

0 

1 

TOTALS 

65 

22 

43 

Analysis  of  the  22  discussions  reveals  that  the 
researchers  also  addressed  external  validity  issues  in  a 
variety  of  different  ways.  References  detailing  ways  in 
which  specific  studies  addressed  internal  validity  are 
provided  in  subsequent  paragraphs.  Secondary  sources  were 
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referenced  as  the  original  designers  of  individual  data 
collection  instruments.  In  other  discussions,  researchers 
generalized  their  results  without  providing  a 
justification.  Some  discussions  clearly  cautioned  readers 
that  the  external  validity  of  instruments  had  not  been 
demonstrated . 

References  to  secondary  sources  were  considered  an 
attempt  to  siabstantiate  the  external  validity  of  individual 
data  collection  instruments.  Eleven  studies  cited  secondary 
sources;  Lucas  (1974) ,  Lucas  (1975b) ,  King  and  Rodriguez 
(1978) ,  Robey  and  Zeller  (1978) ,  Lucas  (1978) ,  Robey  (1979) , 
Baroudi  et  al .  (1986),  Franz  et  al.  (1986),  Tait  and  Vessey 

(1988) ,  and  Cronan  and  Douglas  (1990) .  Appendix  B 
identifies  the  specific  source (s)  cited  by  each  of  these 
studies.  As  noted  earlier,  the  citation  of  secondary 
sources  implies  that  the  individual  instrxaments  were 
validated  in  previous  studies.  Cited  secondary  sources  are 
expected  to  provide  readers  with  information  regarding  the 
external  validity  of  individual  instrviments . 

When  the  secondary  sources  cited  by  Lucas  (1974) , 
Lucas  (1975b),  King  and  Rodriguez  (1978),  Robey  and  Zeller 
(1978),  Robey  (1979),  and  Franz  et  al.  (1986)^  were 
examined,  little  discussion  of  external  validity  was  found. 
The  cited  secondary  sources  that  provided  insufficient 


^Reference  to  secondary  source  of  Sims,  Szilagyi, 
Keller  (1976)  provided  discussion  of  validity  issues. 
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discussion  of  external  validity  were  Lucas  (1973),  Schultz 
and  Slevin  (1975),  Wolek  (1975),  Ein-Dor  and  Segev  (1978), 
Keen  (1981),  and  McFarlan  (1981). 

Secondary  sources  cited  by  Lucas  (1978) ,  Baroudi  et 
al .  (1986),  Franz  et  al .  (1986)^,  Tait  and  Vessey  (1988), 

and  Cronan  and  Douglas  (1990)  directly  discussed  the 
external  validity  of  individual  instruments.  The  cited 
secondary  sources  that  provided  adequate  discussion  of 
external  validity  were  Smith  et  al.  (1969),  Elizur  and 
Guttman  (1976),  Sims  et  al .  (1976),  Bailey  and  Pearson 

(1983),  Ives  et  al .  (1983),  Baroudi  and  Orlikowski  (1988), 

and  Doll  and  Torkzabeh  (1988) . 

Seven  of  the  discussions  about  external  validity 
involved  researchers  generalizing  the  findings  of  their 
studies.  By  having  generalized  findings,  these  researchers 
were  considered  to  have  at  least  implied  that  the  external 
validity  of  their  individual  instruments  had  been 
demonstrated.  Elchesen  (1978),  Mansour  and  Watson  (1980), 
Altmeyer  and  Bozeman  (1981)  ,  Cron  auid  Sobol  (1983)  ,  Bruwer 
(1984),  Decker  and  Plumlee  (1985),  and  Kapsales  (1986)  all 
generalized  findings  resulting  from  their  individual 
instruments.  Cron  and  Sobol  (1983)  were  the  only 
researchers  that  explicitly  discussed  the  external  validity 
of  their  individual  instruments. 


^Reference  to  secondary  source  of  Schultz  and  Slevin 
(1975)  provided  little  discussion  of  validity  issues. 
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six  of  the  discussions  of  external  validity 
acknowledged  deficiencies.  Gallagher  (1974)  and  Franz  and 


Robey  (1986)  stated  that  the  validity  of  their 
questionnaires  had  not  been  established.  Tait  and  Vessey 
(1988)  implied  that  the  validity  of  their  questionnaire  had 
not  been  established.  Edstrom  (1977)  cautioned  against 
generalizing  the  findings  from  his  interviews.  Cron  and 
Sobol  (1983)  and  Aldag  and  Power  (1986)  cautioned  that 
further  studies  involving  their  questionnaires  were 
required. 

Baroudi  et  al .  (1986)  provided  the  only  discussion 

of  actions  taken  which  allegedly  enhanced  the  abilities  to 
generalize  their  results .  Researchers  typically  either 
ignored  external  validity  issues  or  generalized  findings 
without  supporting  discussions. 

3.  Types  of  Validity  Claimed  or  Implied 

Researchers  provided  discussions  about  either  the 
internal  or  external  validity  of  25  of  the  65  individual 
data  collection  instruments.  Specific  types  of  validity 
were  claimed  or  implied  for  only  15  of  the  25  instruments . 
The  remaining  ten  instruments  could  not  be  linked  to  a 
specific  type  of  validity. 

As  previously  elaiborated,  discussions  regarding  six 
instruments  involved  only  references  to  secondary  sources 
that  were  found  to  provide  little  or  no  discussions  of 
validity  issues.  Consequently,  such  discussions  were  not 
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considered  as  having  claimed  or  implied  a  specific  type  of 
validity  for  the  involved  instruments.  Other  discussions 
were  presented  asserting  that  four  instruments  had  not  been 
validated,  again  not  identifying  a  specific  type  of 
validity . 

An  analysis  of  the  discussions  which  claimed  or 
implied  a  specific  type  of  validity  for  the  remaining  15 
instrviments  reveals  that  the  researchers  rarely  explicitly 
claimed  validity.  Explicit  claims  of  validity  were  made  for 
four  instrviments.  In  contrast,  they  implied  specific  types 
of  validity  for  ten  instruments . 

The  discussions  that  explicitly  claimed  validity 
involved  four  of  the  37  studies .  Discussions  identifying  a 
specific  type  of  validity  were  presented  for  five  of  the  65 
instrviments  used.  All  instruments  with  such  discussions 
were  questionnaires . 

Aldag  2md  Power  (1986)  made  claims  of  internal 
validity  for  their  questionnaire.  Baroudi  et  al.  (1986) 
made  claims  of  internal  emd  external  validity,  specifically 
citing  content  and  construct  validity  for  one  of  their 
questionnaires.  Franz  et  al.  (1986)  made  claims  of  internal 
and  external  validity,  specifically  citing  construct  and 
discriminant-convergent  validity  for  one  of  their 
questionnaires.  Cronan  and  Douglas  (1990)  also  made  claims 
of  internal  and  external  validity,  specifically  citing 
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construct,  convergent-discriminant,  and  criterion  validity 
for  their  two  questionnaires. 

An  analysis  of  the  discussions  which  addressed 
validity,  having  excluded  references  to  secondary  sources 
that  provided  little  or  no  information  regarding  validity 
issues,  revealed  that  researchers  implied  specific  types  of 
validity  for  ten  instriiments .  The  internal  validity  of  two 
instruments  were  implied;  individual  questionnaires  used  by 
Lucas  (1978)  and  Franz  and  Robey  (1986) .  The  external 
validity  of  six  instruments  were  implied;  observations 
collected  by  Elchesen  (1978),  questionnaires  administered  by 
Altmeyer  and  Bozeman  (1981) ,  Cron  and  Sobol  (1983) ,  Bruwer 
(1984) ,  and  archival  records  analyzed  by  Decker  and  Plumlee 
(1985) ,  and  Kapsales  (1986) .  Both  the  internal  and  external 
validity  of  two  instruments  were  implied;  a  questionnaire 
administered  by  Mansour  and  Watson  (1980)  and  observations 
collected  by  Durand  et  al.  (1987) . 

4 .  Sources  of  Claimed  or  laplied  Validity 

Five  questionnaires  claimed  a  specific  type  of 
validity;  Aldag  and  Power  (1986),  Baroudi  et  al.  (1986), 
Franz  et  al .  (1986),  and  Cronan  and  Douglas  (1990) .  They 

discussed  the  analyses  of  instrument  measures  as 
justification  for  claiming  specific  types  of  validity. 

Ten  data  collection  instruments  were  implied  to 
possess  a  specific  type  of  validity;  Elchesen  (1978) ,  Lucas 
(1978) ,  Mansour  and  Watson  (1980) ,  Altmeyer  and  Bozeman 
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(1981),  Cron  and  Sobol  (1983),  Bruwer  (1984),  Decker  and 
Plumlee  (1985) ,  Franz  and  Robey  (1986) ,  Kapsales  (1986) , 
and  Durand  et  al.  (1987) .  None  of  the  discussions  which 
involved  these  instruments  identified  the  source  of  the 
validity . 

5 .  Collectively  Addressing  Validity  Issues 

Information  systems  researchers  used  multiple  data 
collection  instruments  in  18  of  the  37  studies.  Fourteen  of 
the  studies  that  involved  multiple  instruments  included 
discussions  that  addressed  validity  issues  in  a  collective 
manner.  Essentially,  the  researchers  addressed  the 
collective  validity  of  the  study,  rather  than  the  validity 
of  individual  instruments . 

Data  were  collected  regarding  discussions  that 
collectively  addressed  either  internal  or  external  validity 
issues.  These  discussions  were  used  to  determine  the 
specific  types  of  validity  that  researchers  implied  or 
claimed  their  studies  demonstrated.  None  of  the  discussions 
identified  the  sources  of  implied  or  claimed  validity. 

Analysis  of  the  14  discussions  which  collectively 
addressed  validity  issues  reveals  researchers  asserted  the 
validity  of  six  studies  had  not  been  esta±)lished;  Swanson 
(1974) ,  Lucas  (1975a) ,  Lucas  (1975b) ,  Schewe  (1976) ,  Neumann 
and  Segev  (1979) ,  and  Robey  (1979) .  A  single  study,  Gallupe 
and  DeSanctis  (1988) ,  included  discussion  that  implied  only 
internal  validity  had  been  collectively  demonstrated.  Three 
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studies  included  discussions  that  implied  only  external 
validity  had  been  collectively  demonstrated;  Hall  and 
Lincoln  (1976) ,  Robey  and  Zeller  (1978) ,  and  Ginzberg 
(1981) .  Another  two  studies.  Powers  and  Dickson  (1973)  and 
Lucas  (1974) ,  claimed  that  only  external  validity  had  been 
collectively  demonstrated.  Two  studies  implied  that  both 
internal  and  external  validity  had  been  collectively 
demonstrated;  Franz  et  al .  (1986)  and  Cronan  and  Douglas 

(1990) .  Additionally,  Cronan  and  Douglas  (1990)  was  the 
only  study  to  collectively  cite  specific  types  of  internal 
validity;  construct  and  convergent  validity. 

Only  two  of  the  37  studies  included  discussions  that 
individually  and  collectively  addressed  validity  issues. 
Franz  et  al .  (1986)  and  Cronan  and  Douglas  (1990)  included 
discussions  regarding  the  validity  of  individual 
instruments .  They  also  provided  discussions  regarding  the 
collective  validity  of  their  studies. 
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V.  CONCLUSIONS 


The  conclusions  that  follow  are  based  upon  key  findings 
presented  in  the  previous  chapter.  The  conclusions  are 
ordered  in  same  sequence  as  the  findings .  Only  summary 
totals  are  provided  in  this  chapter.  References  to  specific 
instruments  and  studies  which  comprise  the  summary  totals 
are  located  with  the  corresponding  findings,  in  the  previous 
chapter . 

A.  DATA  COLLECTION  INSTRUMENTS 
1 .  Selection  of  Instruments 

Questionnaires  were  the  primary  data  collection 
instrument  used  by  researchers.  Questionnaires  were  used  in 
29  of  the  37  studies,  accounting  for  36  of  the  65 
instruments.  This  suggests  that  the  majority  of  the  data 
from  which  researchers  generate  findings  are  collected  with 
questionnaires.  Their  reliance  on  cpjestionnaires  leaves 
researchers  dependent  upon  respondents  to  report  on 
effectiveness  of  information  systems.  Denzin  (1970) 
discussed  threats  to  validity  associated  with  the  use 
questionnaires.  The  obtrusive  nature  of  questionnaires, 
respondent  biases,  and  human  memory  errors  are  likely  to 
result  in  many  respondents  providing  inaccurate  reports  of 
information  systems  effectiveness. 
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Researchers  relied  on  analysis  of  archival  records 
in  a  minority  of  their  studies,  representing  only  15  of  the 
65  instruments.  Information  systems  are,  by  their  very 
nature,  generators  and  repositories  of  archival  records. 

The  archival  records  stored  in  computer  files  provide  an 
unobtrusive  manner  in  which  to  collect  generally  accurate 
data  regarding  information  systems.  Surprisingly  few 
researchers  took  advantage  of  archival  records  as  a  means  of 
studying  information  systems. 

Few  researchers  used  alternative  data  collection 
technicjues,  such  as  interviews  or  observations.  These 
techniques  accounted  for  only  13  of  the  65  instruments . 
Difficulties  associated  with  administering  interviews  and 
collecting  observations  may  account  for  their  less  frequent 
use  in  the  37  studies . 

Interviews  require  direct  person-to-person 
exchanges,  which  are  usually  time  consuming  and  costly  to 
conduct .  Researchers  may  have  opted  to  use  questionnaires 
rather  than  interviews,  since  questionnaire  are  generally 
easier  and  cheaper  to  administer.  Interviews  were 
identified  as  only  9  of  the  65  instrviments .  Interviews  are 
subject  to  many  of  the  data  accuracy  problems  associated 
with  questionnaires.  They  are  obtrusive,  susceptible  to 
personal  biases,  and  prone  to  human  memory  errors. 

Observations  are  difficult  to  collect  outside  the 
laboratory;  consequently  they  are  seldom  used  to  measure 
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operational  information  systems.  Observations  were  the 
bases  for  only  four  of  the  65  instruments.  Observations 
provide  accurate  data,  yet  researchers  appear  reluctant  to 
develop  experimental  designs  that  use  this  data  collection 
technique . 

2 .  Impact  upon  Triangulation 

Multiple  data  collection  instruments  were  used  in 
fewer  than  one-half  of  the  studies'  research  designs.  Most 
researchers  were  precluded  from  triangulating  measures 
because  19  of  the  37  studies  involved  research  designs  that 
used  only  one  instrtiment .  A  greater  use  of  multiple 
instrximents  would  have  provided  more  opportunities  for 
triangulation,  enhancing  the  validity  of  data  collected  in 
the  studies . 

3.  Problems  with  Designs  of  Questionnaires 

The  researchers  demonstrated  a  reliance  on  ordinal 
scales,  using  them  in  30  of  the  31  questionnaires.  Only 
five  studies  explicitly  identified  the  questionnaires  as 
Likert-type.  Because  the  researchers  consistently  discussed 
the  use  of  five,  seven,  and  ten  point  scales,  most  of  the 
questionnaires  were  pres\imedly  Likert-type.  Galletta  and 
Lederer  (1987)  cautioned  that  ntunerous  problems  are 
associated  with  Likert-type  scales.  These  problems  include 
scale  units  and  origins  (anchoring) ,  derivation  of  a  score 
by  adding  heterogeneous  item  scores,  and  a  problem  similar 
to  that  with  the  user  involvement  measure,  and  the  use  of 
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parametric  statistics.  Since  the  majority  of  the 
discipline's  studies  appear  to  use  questionnaires  involving 
Likert-type  designs,  information  systems  research  is 
vulnerable  to  these  problems. 

Researchers  failed  to  state,  in  25  of  36 
questionnaires,  whether  they  were  of  original  design  or 
adapted  from  previous  studies.  Not  knowing  the  source  of  an 
instrument's  design,  impedes  evaluation  of  its  validity. 

The  failure  to  cite  original  design  sources  wrongly 
precluded  readers  from  referencing  discussions  of  validity 
issues  which  may  have  appeared  in  previous  studies. 

4 .  Use  of  Con^uter  Records 

Analyses  of  existing  computer  records  accounted  for 
13  of  the  65  instruments.  Few  researchers  are  exploiting 
computer  records,  which  might  already  reside  on  the 
investigated  information  systems,  as  the  bases  of  their 
data.  Rather  than  using  objective  measures  provided  by 
archival  records,  they  are  primarily  turning  to  a  secondary 
subjective  measures,  as  provided  by  hvunan  respondents. 

5 .  Ambiguous  Designs  of  Interviews 

Sufficient  details  were  provided  to  make 

determinations  regarding  schedule  and  structure  for  only 
three  of  the  nine  interviews  conducted.  None  of  the 
information  systems  researchers  provided  sufficient 
discussion  to  develop  an  understanding  of  the  manner  in 
which  the  interviews  were  conducted  and  the  questions 
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presented.  Researchers  did  little  more  than  identify  that 
interviews  were  conducted,  thus  making  interpretations  of 
conduct  and  content  virtually  impossible. 

6 .  Obtrusive  Role  of  Observers 

Only  four  of  the  37  studies  involved  collection  of 
data  by  observers .  The  researchers  used  participant 
observers  in  three  studies  and  complete  observers  in  one 
study.  Participant  observation  is  an  obtrusive  form  of  data 
collection  and  complete  observation  is  an  unobtrusive  form 
of  data  collection .  The  more  frequent  reliance  upon 
participant  observation  demonstrates  a  possible  tendency  for 
researcher  to  relieve  more  on  obtrusive  rather  than 
unobtrusive  forms  of  observation. 

Despite  having  used  obtrusive  forms  of  observation, 
none  of  the  researchers  assessed  impacts  of  observers  on 
studied  information  systems.  Researchers  failed  to 
recognize  that  the  introduction  of  observers  impacted 
recorded  measures .  Although  unobtrusive  observation  of 
information  systems  is  clearly  desiredDle  over  obtrusive 
observation,  it  is  difficult  for  the  observer  to  remain 
concealed  and  can  be  very  time  consuming  when  conducted 
outside  the  le±)oratory.  This  may  explain  why  the  majority 
of  researchers  who  used  observations  selected  the  obtrusive 
form  of  participant  observation. 


64 


B.  FAILURE  TO  RECOGNIZE  MEASURES  OF  ATTITUDE  OR  BELIEF 

Marshall  and  Rossman  (1989)  describes  questionnaires  as 
instriiments  that  measure  sets  of  attitudes  or  beliefs. 
Interviews  also  solicit  the  respondent  to  express  attitudes 
or  beliefs  regarding  phenomena  of  interest.  Researchers  who 
use  questionnaires  or  interviews  are,  by  the  very  nature  of 
these  instruments,  measuring  the  attitudes  or  beliefs  of 
respondents . 

The  researchers  used  a  total  of  45  questionnaires  and 
interviews .  Only  27  of  the  45  questionnaires  and  interviews 
were  acknowledged  to  have  measured  attitudes  or  beliefs. 

Many  researchers  did  not  acknowledge  or  were  unaware  that 
they  measured  attitudes  or  briefs  with  questionnaires  and 
interviews . 

Many  researchers  purported  that  they  collected  other 
types  of  measures  with  questionnaires  and  interviews .  The 
45  questionnaires  and  interviews  used  by  the  researchers 
were  purported  to  provide  37  measures  of  system 
characteristics,  behavior,  and  situational  characteristics. 
Many  researchers  erroneously  believed  questionnaires  and 
interviews  measure  system  characteristic,  behavior,  and 
situational  characteristics;  when  in  fact,  these  instruments 
exclusively  measure  attitudes  or  beliefs. 
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C.  ADMINISTXULTZON  OF  DATA  COLLECTION  INSTRUMENTS 

1 .  Sandies  of  Convanience 

All  37  studies  appear  to  involve  seunples  selected  on 

bases  of  convenience.  The  ability  to  generalize  findings 

becomes  questionable  when  seimple  are  selected  on  the  bases 

of  convenience .  The  characteristics  of  such  samples  may  not 

accurately  reflect  the  population  they  are  supposed  to 

represent.  Cochran  (1953)  contends, 

"The  people  from  whom  data  are  obtained,  the  sauntiple,  are 
of  interest  only  insofar  as  the  data  tell  us  something 
zdjout  some  larger  group  of  people  whom  statisticians 
call  the  population  or  universe.  Further,  results 
obtained  from  a  sample  can  be  extended  to  a  larger 
population  with  logical  soundness  only  if  the  saunple  is, 
in  a  certain  technical  sense,  a  probability  sample  drawn 
from  that  population." 

Samples  selected  on  bases  of  convenience  are  not 
necessarily  a  probability  sample  drawn  from  the  populations 
under  study.  Without  auiy  indications  that  selected  samples 
represent  the  studied  populations,  researchers  are 
generalizing  the  findings  on  their  studies. 

2 .  Practitioners  and  Users  as  Respondents 

A  total  of  49  questionnaires,  interviews,  and 
observations  were  used  as  data  collection  instruments . 

Humans  either  responded  to  or  were  the  subjects  observed  for 
all  of  these  instruments.  Practitioners  and  users  of 
information  systems  were  the  focus  of  43  of  these  49 
instruments.  The  remaining  6  instruments  involved  students. 
The  high  proportion  of  studies  involving  practitioners  and 
users  of  information  systems,  rather  than  college  students, 
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enhanced  the  eibility  to  generalize  the  findings  of  the 
studies  onto  operational  situations. 

3 .  Insufficient  Information  Concerning  San^le  Size 
The  researchers  failed  to  provide  information 

clearly  identifying  the  number  of  samples,  duration  of 
sampling,  or  frequency  of  collections  for  27  of  the  65 
instruments.  The  eibsence  of  such  information  prevents 
readers  from  fully  assessing  how  well  samples  represented 
the  studied  populations. 

4 .  Insufficient  Information  Concerning  Response  Rates 
The  questionnaires  and  interviews  used  in  the  37 

studies  all  relied  upon  respondent  participation.  The 
representativeness  of  saunples  collected  with  these 
instruments  is  affected  by  the  response  rates  achieved.  The 
researchers  failed  to  provide  information  regarding  the 
response  rates  achieved  for  20  of  the  40  questionnaires  and 
interviews .  The  eJosence  of  such  information  prevents 
evaluation  of  how  well  the  samples  represented  the  studied 
populations . 

5 .  Distribution  of  Questionnaires  by  Mail 

The  researchers  failed  to  provided  information 
identifying  how  18  of  the  36  questionnaires  used  were 
distributed.  Mail  was  the  most  common  method  of 
distribution,  used  for  15  of  the  36  questionnaires.  The 
researchers  that  mailed  questionnaires  were  vulnerable  to 
persons  other  than  the  identified  respondents  having 


completed  the  c[ue3tionnaires .  Without  face-to-face 
interaction,  it  is  difficult  to  know  who  completes  a 
questionnaire.  Because  the  researchers  do  not  know  for 
certain  the  identities  of  the  respondents,  the 
representative  of  the  samples  becomes  suspect. 

6 .  Few  Rasearchers  Discuss  Hon-Respondents 
The  researchers  failed  to  discuss  differences  between 
respondents  and  non-respondents  for  30  of  the  40  instruments 
involving  questionnaires  or  interviews.  The  failure  to 
identify  differences  precluded  readers  from  assessing 
whether  the  researchers  had  addressed  issues  regarding 
respondent  biases.  The  absence  of  discussions  of 
differences  between  respondents  and  non-respondents  raised 
doubts  concerning  the  representativeness  of  almost  all 
samples . 

D.  HANDLING  OF  VALIDITY  ISSUES 

1 .  Few  Researchers  Address  Internal  Validity  Issues 

The  researchers  acknowledged  deficiencies  regarding 
the  internal  validity  of  six  of  the  65  data  instruments. 

The  acknowledgements  of  such  deficiencies  indicates  that  the 
researchers  responsible  for  the  six  involved  studies  were 
aware  and  sensitive  to  internal  validity  issues. 

The  researchers  asserted  another  eight  of  the  65 
instruments  demonstrated  internal  validity.  The  assertions 
of  internal  validity  indicates  that  the  researchers 
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responsible  for  the  eight  involved  studies  were  also  aware 
and  sensitive  to  internal  validity  issues. 

Sufficient  discussions  of  internal  validity  were  not 
presented  for  the  overwhelming  majority  of  instruments.  The 
researchers  failed  to  present  discussions  of  internal 
validity  for  46  of  the  65  instrxjments .  Their  discussions 
for  the  remaining  five  instruments  cited  secondary  sources 
which  provided  little  evidence  supporting  internal  validity. 

In  all,  51  of  the  65  instruments  used  in  the  37 
studies  were  not  accompanied  by  adequate  discussions  of 
internal  validity.  Instriiments  are  not  considered  to  have 
demonstrated  validity  unless  sufficient  evidence  is 
presented  to  support  such  claims.  None  of  these  51 
instruments  can  be  considered  to  have  demonstrated  internal 
validity.  All  of  the  researchers  who  used  these  instruments 
based  their  findings  upon  measures  that  have  not 
demonstrated  internal  validity.  This  raises  a  question  as 
to  whether  these  researchers,  most  of  those  involved  in  the 
37  studies,  ignored  internal  validity  issues  or  were  simply 
not  aware  of  them. 

2 .  Few  Researchers  Address  External  Validity  Issues 

The  researchers  acknowledged  deficiencies  regarding 
the  external  validity  of  six  of  the  65  instruments.  The 
acknowledgements  of  such  deficiencies  indicates  that  the 
researchers  responsible  for  the  six  involved  studies  were 
aware  and  sensitive  to  external  validity  issues. 
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The  researchers  responsible  for  four  studies 
presented  or  cited  discussions  which  asserted  that  their 
instruments  had  demonstrated  external  validity.  Clearly, 
they  were  aware  and  sensitive  to  external  validity  issues. 

Without  having  provided  discussions  of  external 
validity  issues,  the  researchers  generalized  the  findings  of 
studies  which  were  based  upon  data  from  six  of  the  65 
instruments.  The  absence  of  explicit  discussions  of 
external  validity  issues  indicates  that  these  researchers, 
responsible  for  seven  of  the  studies,  either  ignored  or  were 
unaware  of  external  validity  issues. 

Sufficient  discussions  of  external  validity  were  not 
presented  for  the  overwhelming  majority  of  data  collection 
instruments.  The  researchers  failed  to  present  discussions 
of  external  validity  for  another  43  of  the  65  instruments. 
Their  discussions  regarding  the  remaining  six  instrxjments 
cited  secondary  sources  which  provided  little  evidence 
supporting  external  validity. 

In  all,  55  of  the  65  instruments  used  in  the  37 
studies  were  not  accompanied  with  adequate  discussions  of 
external  validity.  None  of  these  55  instruments  can  be 
considered  to  have  demonstrated  external  validity.  All  of 
the  researchers  who  used  these  instruments  based  their 
findings  upon  measures  that  have  not  demonstrated  external 
validity.  This  raises  a  question  as  to  whether  these 
researchers,  most  of  those  involved  in  the  37  analyzed 
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studies,  ignored  external  validity  issues  or  were  simply  not 
aware  of  them. 

3 .  Few  Researchers  Identify  Specific  Types  of 
Validity 

Researchers  claimed  or  implied  that  15  instrviments 
demonstrated  validity.  Specific  types  of  validity  were 
identified  for  five  instrviments  that  were  claimed  to  have 
demonstrated  validity.  The  four  studies,  which  used  these 
instruments,  identified  combinations  of  the  following 
specific  types  of  validity:  internal,  external,  content, 
construct,  discriminant -convergent,  and  criterion.  These 
were  the  only  researchers  who  comprehensively  confronted 
validity  issues,  having  both  claimed  and  identified  specific 
types  of  validity.  These  researchers  were  aware  of  the 
different  types  of  validity. 

The  remaining  ten  data  collection  instruments  were 
implied  to  have  demonstrated  validity.  Their  studies 
included  no  discussions  which  directly  identified  specific 
types  of  validity,  rather  they  implied  individual 
instruments  had  demonstrated  internal  and  or  external 
validity.  Readers  can  only  guess  whether  these  researchers 
were  aware  of  the  different  types  of  validity. 
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4 .  Pew  Researchers  Identify  the  Sources  of  Claimed  or 

In^lied  Validity 

Researchers  claimed  or  implied  that  15  instrviments 
demonstrated  validity.  Researchers  responsible  for  the  four 
studies,  which  claimed  that  five  instruments  demonstrated 
validity,  identified  analyses  of  instrument  measures  as 
their  sources  of  validity.  These  researchers  openly  stated 
that  the  measures  from  their  individual  instrviments  served 
as  the  sources  for  claims  of  validity.  Readers  can 
investigate  these  sources  to  sxibstantiate  the  claims  of 
validity,  enhancing  the  credibility  of  the  studies. 

The  remaining  ten  data  collection  instruments  were 
implied  to  have  demonstrated  validity.  None  of  the  ten 
studies  involving  these  instruments  identified  sources  of 
validity .  The  absence  of  cited  sources  precludes  readers 
from  substantiating  the  implied  validity  of  these 
instrviments,  detracting  from  the  credibility  of  the  studies. 

5.  Inadequacy  of  Collectively  Addressing  Validity 

Issues 

Eighteen  studies  used  multiple  data  collection 
instriiments .  In  14  of  these  studies,  researchers  presented 
discussions  that  addressed  validity  issues  in  a  collective 
manner.  Such  discussions  provided  no  information  regarding 
the  validity  of  individual  instruments,  rather  they  solely 
addressed  the  collective  validity  of  the  entire  studies. 
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Although  discussions  regarding  collective  validity  were 
interesting  to  read,  they  were  essentially  meaningless  in 
the  cibsence  of  discussions  that  substantiated  the  validity 
individual  instruments .  Without  having  presented  evidence 
that  the  individual  instruments  have  demonstrated  validity, 
researcher  can  not  reasoneibly  expect  to  claim  or  imply 
collective  validity  for  entire  studies. 

Many  researchers  appear  to  have  mistakenly  believed 
that  discussions  of  only  collective  validity  adequately 
addressed  validity  issues .  Researchers  often  presented 
discussions  of  only  collective  validity  in  their  studies . 
Only  two  of  the  14  studies  included  discussions  of  both 
individual  and  collective  validity.  The  cibsence  of 
discussions  substantiating  the  validity  of  individual 
instruments,  necessitates  that  claims  of  collective  validity 
be  discounted  for  the  12  studies  that  included  discussions 
of  only  collective  validity. 

S .  RECOMMENDATIONS 

A  minority  of  researchers  addressed  validity  issues. 
Discussions  regarding  the  validity  of  data  collection 
instruments  were  generally  superficial.  Rarely  did 
researchers  demonstrate  awareness  or  sensitivity  to  validity 
issues .  Most  researchers  appear  to  have  expected 
unquestioned  acceptance  of  their  findings,  since  readers 
were  provided  with  relatively  few  discussions  of  validity. 
Such  expectations  are  inappropriate  in  scientific  research. 
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The  failure  to  discuss  validity  issues  detracts  from  the 
credibility  of  research  involving  the  information  systems 
discipline . 

An  emphasis  on  validity  issues  should  be  encouraged 
throughout  the  information  systems  discipline.  Students, 
practitioners,  and  researchers  must  be  made  acutely  aware 
and  sensitive  to  the  concept  of  validity.  Implementation  of 
the  following  suggestions  would  heighten  awareness  and 
sensitivity : 

1 .  Students  should  receive  exposure  to  the  concept  of 
validity  as  part  of  their  graduate  school  instruction. 

2 .  Practitioners  should  implement  the  recommendations 
resulting  from  only  studies  that  discussed  and 
demonstrated  validity. 

3.  Professional  journals  pertaining  to  the  information 
systems  discipline  should  require  that  all  articles 
submitted  by  researchers  include  discussions  of 
validity  issues. 

Information  systems  researchers  have  adopted  the  data 
collection  instruments  commonly  found  in  social  science 
research,  yet  have  largely  failed  to  face-up  to  the  validity 
issues  that  inherently  arise  with  their  use.  Social 
scientists  have  long  understood  and  addressed  validity 
issues .  Information  systems  researchers  should  confront 
validity  issues  in  the  same  manner  as  social  scientists. 

They  must  become  aware  of  the  threats  to  validity  and 
develop  experimental  designs  which  enhance  both  internal  and 
external  validity.  They  must  clearly  and  comprehensively 


74 


discuss  validity  issues,  adding  credibility  to  information 
systems  research. 
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APPENDIX  B 


DETAILED  ANALYSIS  OF  STUDIES 


Colton,  K.  W.,  "Computers  and  Police:  Patterns  of  Success 
and  Failure,"  Sloan  Management  Review,  Vol .  14,  No.  2, 
(Winter  1972-73),  pp.  75-98. 


Description  of  Svatem  Studied:  Information  systems  used  by 
police  departments.  Applications  included  police  patrol  and 
daily  operations;  investigative  analysis;  administration  and 
management;  and  program,  planning,  and  evaluation. 

Data  Collection  Technioue(s)  Used:  All  respondents 
completed  questionnaire .  Selected  respondents  were  later 
interviewed. 

Characteristics  of  Measuring  Instruments  Used:  Specific 
characteristics  of  the  measuring  instruments  were  not 
explicitly  discussed.  The  survey  was  devised  by  the 
International  City  Management  Association,  appearing  not  to 
have  complied  with  an  established  format.  No  formal 
questionnaire  was  administered,  but  an  effort  was  made  to 
ask  a  common  set  of  questions- — an  unstructured  and  scheduled 
interview. 

Obi  active  (a)  of  Mea0”^»"«»"t  Procaaa:  Questionnaires  were 
aimed  at  documenting  the  growth,  extent,  and  impact  of 
computers  used  by  police  departments.  Interviews  were  aimed 
at  evaluating  questions  on  impact,  effectiveness,  and 
success.  Data  collected  were  not  specifically  identified  as 
measures  of  attitude,  opinion,  behavior,  or  etc.. 

Participants  in  Maasuremant  Procaee:  Police  chiefs  and  data 
processing  managers  of  police  departments  in  cities  with 
populations  of  over  25,000. 

Distribution  of  Maaeurinq  Instruments:  Questionnaires  were 
mailed  to  all  police  departments  in  cities  with  populations 
of  over  50,000  and  to  25%  of  cities  with  populations  between 
25,000  and  50,000.  A  total  of  498  questionnaires  were 
mailed  and  37  6  responses  were  received  (75.5%)  .  Interviews 
were  personally  conducted  at  14  police  departments.  Visits 
of  3  to  4  days  were  conducted  at  9  police  departments  and 
brief  contacts  were  made  in  5  other  departments .  A  total  of 
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143  interviews  were  conducted  with  law  enforcement 
personnel . 


Corrective  Action  taken  to  Accotant  for  Non-participants : 
Second  mailings  to  cities  that  failed  to  respond  were  made. 

Identification  of  Differences  between  Participants  and  Non- 
participanta :  Not  discussed. 

Method  Used  to  Analyze  Data:  Method  of  data  analysis  not 
stated. 

Issue  of  Validity  Addressed:  Not  discussed. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative,  Metric,  Interval, 
Frequency,  and  Ordinal. 
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Powers,  R.  F.  and  G.  W.  Dickson,  "MIS  Project  Management: 
Myths,  Opinions  and  Reality,"  California  Management  Review, 
Vol.  15,  No.  3,  (1973),  pp.  147-156. 


Description  of  Svstcun  Studied:  Twenty  management 
information  systems  used  in  the  manufacturing,  wholesale  and 
retail  trade,  transportation,  finance,  utility,  and 
commodity  merchandising  and  processing  businesses . 

Data  Collection  Technique (s)  Deed:  Questionnaire  for 
initially  identifying  factors  affecting  MIS  project  success. 
Sxibsequent  interview  and  cjuestionnaire  to  research 
correlates  of  success  with  MIS  projects. 

Characteristics  of  Measuring  instiiiinftnts  Used:  The 

questionnaire  used  to  initially  identify  factors  affecting 
MIS  project  success  required  the  respondent  to  rank  the 
importance  of  35  factors,  as  selected  by  25  MIS  experts,  on 
a  five  point  scale.  Only  the  average  time  spent  during  each 
interview  was  provided,  9  hours  per  firm.  No  information 
was  provided  pertaining  to  the  second  questionnaire,  beyond 
the  fact  that  it  was  pretested. 

Objective  (s)  of  Measuranent  Process;  Determine  what 
organizational  and  procedural  factors  are  correlates  of 
success  with  MIS  projects.  The  questionnaire  used  to 
initially  identify  factors  affecting  MIS  project  success  was 
acknowledged  as  measuring  beliefs.  Data  collected  in 
interviews  and  with  second  questionnaire  were  not 
specifically  identified  as  measuring  beliefs,  attitudes, 
behavior,  or  etc.. 

Participants  in  Measurement  Proceee:  Initial  questionnaire 
was  given  to  MIS  experts  attending  the  Founding  Conference 
of  the  Society  for  Management  Information  Systems  at  the 
University  of  Minnesota  in  1969.  The  interviews  and  second 
questionnaires  were  used  to  study  10  firms,  with  each 
providing  2  MIS  projects,  in  the  Minneapolis  and  St.  Paul 
area.  The  respondents  within  the  firms  were  directors  of 
MIS,  project  leaders,  computer  operating  personnel,  and 
management  users . 

Distribution  of  Measuring  Instruments :  The  initial 
questionnaire  was  given  to  the  respondents.  The  interview 
and  second  questionnaire  was  conducted  on-site,  at  the 
firms.  Response  rates  were  not  discussed. 

Corrective  Action  taken  to  Accoiant  for  Non-participants : 

Not  discussed. 
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Identification  of  Differences  between  Participants  and  Non- 
participants  :  Not  discussed. 

Method  Deed  to  Analyze  Data:  Data  were  analyzed  using  non- 
parcimetric  statistical  techniques,  specifically  Kendall's 
tau  statistic. 

Issue  of  Validity  Addressed:  "Kendall's  tau  statistic  was 
chosen  for  the  data  analysis  because  of  its  statistical 
properties  which  facilitated  testing  the  various  hypotheses 
for  significance."  The  ability  to  generalize  findings  was 
briefly  discussed  (External  validity) . 

Type  of  Validity  Claimed:  External  validity,  the  authors 
stated  that  in  their  opinions  their  findings  could  be 
generalized  to  MIS  projects. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative,  Metric,  Interval,  and 
Frequency . 
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ISBue  of  Validity  Addressed:  The  author  discussed  extension 
of  study:  "The  research  techniques  and  instruments  in  this 
study  can  be  used  by  any  organization  to  evaluate  its 
information  systems.  This  study  has  demonstrated  an 
association  between  systems  quality  and  user  reactions  to 
information  systems....  Though  causality  has  not  been 
demonstrated,  a  priori  arguments  and  data  support  the  goals 
of  developing  favorc±>le  user  attitudes  and  high  quality 
systems  to  encourage  use  of  information  systems." 

Type  of  Validity  Claimed:  External  validity  through 
argument  of  extension. 

Source  of  Validity  Claimed:  A  priori  arguments  and  data. 
Type  of  Data  Recorded:  Ordinal. 

Secondary  Source (s^  Cited:  Lucas  (1973)  is  where  the 
original  questionnaire  was  presented,  however,  its  validity 
wa?  not  clearly  estcJalished- 
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Swanson,  E.  B.,  "Management  Information  Systems: 
Appreciation  and  Involvement,"  Management  Science,  Vol .  21, 
No.  2,  (October  1974),  pp.  178-188. 


Description  of  ! 

Svstem  Studied:  An  activitv  reoortina  svste 

(MIS)  used  by  a 
international  m; 
The  system  gath« 
activity  of  the 

200  employee  department  of  a  large 
■inuf  acturer  of  complex  electronic  equipment . 
jrs  data  on  the  planned  and  actual  work 
department's  members,  and  makes  it  availabl 

to  manaaement  ot 

1  a  "need  to  know"  basis. 

unaraccerxsrxcs 

oz  neasurxna  xnsrzumenzs  usea:  computer 

to  be  equal.  T1 
ouestionnaire  c< 

le  respondent  checked  an  interval .  The 
ansisted  of  a  set  of  10  items,  based  upon  a 

Obiectiva(s)  of 

Measurement  Process:  Archival  data  used  to 

measure  system  x 
constructed  to  < 

jsage .  The  research  questionnaire  was 
sbtain  indicators  of  MIS  appreciation  and  a 

priori  involvem< 

jnt .  Data  collected  in  questionnaire  was  no 

specifically  id< 
behavior,  or  etc 

sntified  as  measuring  beliefs,  attitudes. 

1  •  •iM  4 

Not  discussed. 

Identification  of  Differences  between  Parbicipemts  and  Non- 
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Issue  of  Validifcy  Addressed:  "It  should  be  noted,  however, 
that  the  direction  of  the  original  causal  argioment  has  not 
been  tested. . .  no  controlled  experimentation  was  undertaken 
in  this  study....  In  short,  there  is  no  beginning  for  a 
causal  argument."  Later,  "The  single  case  presented  here 
does  not  permit  effective  generalization." 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal  and  frequency  (using  an 
ordinal  scale) . 
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Gallagher,  C.  A.,  "Perceptions  of  the  Value  of  a  Management 
Information  System,"  Academy  of  Management  Journal,  Vol .  17, 
No.  1,  (March  1974),  pp .  46-55. 


Description  of  System  Studied:  A  management  information 
system  used  for  cost  accounting.  It  was  developed  in 
1967/68  and  used  by  a  company  employing  1,800  people,  of 
whom  375  were  management  personnel. 

Data  Collection  Technique (s)  Used:  Questionnaire . 

Characteristics  of  Measuring  Instruments  Used: 

Questionnaire  consists  of  two  parts.  The  first  part 
solicited  an  opinion  expressed  in  dollar  values.  The  second 
part  solicited  an  opinion  reported  by  scoring  fifteen 
bipolar  adjective  pairs  on  a  seven-point  scale. 

Objective (s)  of  Measurement  Process:  The  study  was  used  to 
determine  the  value  of  an  existing  set  of  MIS  reports.  The 
study  measures  user  perceptions . 

Participants  in  Measurement  Process:  Mamagers  using  reports 
generated  by  the  system. 

Distribution  of  Measuring  Instruments;  The  questionnaire 
was  mailed  to  103  managers,  of  which  75  usable  replies  were 
received. 

Corrective  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Semantic  differential  results 
and  correlation  between  value  measures  were  used  to  analyze 
the  data . 

Issue  of  Validity  Addressed:  "The  principal  challenge  lies 
in  designing  questions  which  will  minimize  the  effects  of 
bias  and  inaccuracy  in  user  perceptions ....  This  study 
included  one  test  of  a  new  methodology.  The  results  of  a 
single  test  cannot  establish  the  validity  of  a  methodology, 
although  they  can  discredit  all  or  part  of  the 
methodology....  A  question  may  be  raised  regarding  the  use 
of  hypothetical  questions .  Do  the  answers  to  a  hypothetical 
question  bear  any  relationship  to  those  that  would  be 
obtained  in  an  actual  situation?  Surely  the  answers  would 
not  be  identical." 
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Type  of  Validity  Claimed:  None 

Source  of  Validity  Claimed:  Study  identified  lack  of 
validity . 

Type  of  Data  Recorded:  Interval  and  ordinal. 
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Lucas,  H.  C.,  Jr.,  "Performance  and  The  Use  of  an 
Information  System,"  Management  Science,  Vol .  21,  No.  8, 
(April  1975a),  pp.  908-919. 


One  follow-up. 

Identification  of  Differences  between  Participants  and  Non¬ 


participants  :  Not  discussed. 


Method  Used  to  Analyze  Data:  Step-wise  multiple  regression 
analysis  was  used  to  analyze  the  data. 

Issue  of  Validity  Addressed:  "It  is  difficult  to  draw 
conclusions  about  causality  from  cross-sectional  studies. 
Both  ladDoratory  studies  with  their  superior  controls  and 
field  studies  are  needed  to  confirm  and  expand  the  findings 
reported  here . " 

Type  of  Validity  Claimed: 


None . 


Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal  and  other  unknown. 
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Lucas,  H.  C.,  Jr.,  "The  Use  of  Accounting  Information 
System,  Action  and  Organizational  Performance, "  Accounting 
Review,  (October  1975b),  pp.  735-746. 


Not  discussed. 


Identification  of  Differences  between  Participants  and  Non¬ 


participants  :  Not  discussed. 


Method  Used  to  Analyze  Data:  Step-wise  multiple  regression 
was  used  to  analyze  the  data  in  the  study. 
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Issue  of  Validity  Addressed:  Factor  analysis  and 
correlation  coefficients  were  used  to  determine  if  items 
could  be  combined  in  a  single  scale  during  the  development 
of  the  questionnaire.  "In  a  field  study  it  is  difficult  to 
obtain  good  measures  for  operationalized  variadsles  and  to 
control  for  confounding  variables .  The  lack  of  extremely 
strong  results  can  be  partially  attributed  to  errors  in 
measuring  the  varieibles  which  were  included  in  the 
research . " 

Type  of  Validity  Claimed :  None . 

Source  of  Validity  Claimed:  None. 

Type  of  Data  Recorded:  Ordinal  and  unknown  other (s). 

Secondary  Source (s)  Cited:  Lucas  (1973)  is  where  the 
original  questionnaire  was  presented,  however,  its  validity 
\  :  not  clearly  established. 
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Hall  P.  G.  and  T.  J.  Lincoln,  "A  Cost  Effectiveness  Study  of 
Production  and  Engineering  Systems  within  a  Tightly 
Integrated  Manufacturing  Environment, "  Management 
Datamatics,  Vol .  5,  No.  6,  (1976),  pp.  262-274. 


Description  of  Svaten  Studied:  Production  and  engineering 
computer  systems  in  the  Commercial  Aircraft  Division  of 
British  Aircraft  Corporation.  Applications  involving 
product  definition,  work  in  progress,  and  supplies 
management  were  performed  on  these  systems . 

Data  Collection  Technique (a)  Deed:  Participant  observation 
and  interviews . 

Characteriatica  of  Measuring  Inatruaenta  Used:  The  study 
began  with  a  two  days  of  observing  the  system  in  question 
and  interviewing  the  senior  systems  analyst .  One  primary 
user  of  each  system  was  interviewed  for  1  to  2  hours .  User 
management  was  interviewed  after  receiving  documentation 
resulting  from  the  primary  user  interview.  It  was  made 
clear  that  the  user  management  interview  would  form  the 
basis  for  further  discussions  with  senior  management.  A 
final  presentation  and  interview  of  senior  management  was 
conducted  prior  to  presentation  of  results  to  the  company's 
board. 

Object ive (s)  of  Measurement  Proceae:  Interview  with  the 
senior  system  analyst  provided  computer  cost  and  background 
information  regarding  system.  Interview  with  primary  users 
identified  user  costs  and  the  impact  of  the  system  on  the 
user's  department.  Interview  of  user  management  provided 
costs  and  benefits  information.  Presentation  and  interview 
of  senior  management  was  aimed  at  informing  management  of 
results  and  ironing  out  any  difficulties  or  uncertainties 
which  still  existed.  Data  collected  were  not  specifically 
identified  as  measurements  of  attitudes,  opinion,  behavior, 
or  etc . . 

Participants  in  Measurement  Process:  Senior  systems 
analysts,  one  primary  user  of  each  system  (as  identified  by 
its  senior  systems  analyst),  user  management,  and  senior 
management . 

Distribution  of  Measuring  Instruments :  All  interviews 
appear  to  have  been  conducted  in  person . 

Corrective  Action  taken  to  Account  for  Non-pairticipants : 

Not  discussed. 


104 


Identification  of  Differences  betwe«n  Participants  and  Non- 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Method  of  data  analysis  not 
stated. 

Issue  of  Validity  Addressed:  Essentially  stated  that  other 
companies  should  conduct  similar  studies  and  generalizes  its 
findings . 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 


Type  of  Data  Recorded:  Interval,  metrics,  and  unknown 
others . 


Schewe,  C.  D.,  "The  Management  Information  System  User:  An 
Exploratory  Behavioral  Analysis,"  Academy  of  Management 
iJournal,  Vol .  19,  No.  4,  (December  1976),  pp.  577-590. 


Description  of  System  Studied:  Eight  batch  and  two 
interactive  systems  of  ten  food  processing  firms  in  three 
midwestern  states . 

Data  Collection  TechniQue(s)  Used:  Questionnaire  and 
possible  analysis  of  existing  records  eind  docioments. 

Characteristics  of  Measuring  Instruments  Used: 

Questionnaire  used  a  five-point,  bipolar  scale.  No  details 
regarding  how  system  usage  (monthly  requests)  were  measured. 

Objective fs)  of  Measurement  Process:  "This  study  focused  on 
attitudes  toward  use  of  the  information  system."  "System 
usage  in  this  study  was  measured  by  the  number  of  monthly 
requests  that  a  manager/ system  user  made  for  additional 
information."  "A  behavioral  study  utilizing  am  attitudinal 
model  explored  the  relationships  between  MIS  users' 
perceptions  of  their  computer  system,  perceived  variaQ:>les 
exogenous  to  the  system,  attitudes,  amd  system  usage." 

Participanta  in  Procaaa:  Middle  managers  (41 

batch  system  users  and  38  interactive  system  users) . 

Diatribution  of  Measuring  •  Not  discussed. 

Corrective  Action  taken  to  Account  for  Non-participanta ; 

Not  discussed. 

Identification  of  Differencea  between  Participants  and  Non- 
participanta  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Data  were  analyzed  by  the  use 
of  step-wise  regression. 

Issue  of  Validity  Addressed:  "This  study  was  exploratory, 
and  the  attitude  variables  studied  are  not  completely 
comprehensive  of  the  system  user's  attitude  system  toward 
usage  of  the  MIS." 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal. 
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Edstrom,  A.,  "User  Influence  and  the  Success  of  MIS 
Projects:  A  Contingency  Approach,"  Human  Relations,  Vol. 

30,  No.  7,  (1977),  pp.  589-607. 


Deacription  of  System  Studied:  Systems  designed  to  support 
management  decision-making,  selected  from  sixteen  companies. 
All  companies  were  located  in  the  Paris  region  of  France . 

The  companies  were  involved  in  manufacturing, 
transportation,  and  service  industries. 

Data  Collection  Technique (a)  Uaed:  Interviews. 

Charactariatica  of  Measuring  Instruments  Used:  Structured 
interviews  with  four  key  players  in  each  organization.  The 
interview  questions  have  been  of  both  the  open  ended  and 
fixed  alternative  answer  kind.  A  seven  point  rating  scale 
was  used. 

Objective (s)  of  Measurement  Process;  "It  should  be  pointed 
out,  however,  that  the  measures  of  outcomes  are  perceptual 
measures  rather  than  assessments  based  on  objective 
criteria."  The  data  were  measures  of  perceptions. 

Participants  in  Measurement  Process:  System  development 
mauiager,  functional  manager,  project  manager,  and  a  user. 

Distribution  of  Measuring  •  Personal  interviews. 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analvae  Data:  Spearmam-rank  correlation 
coefficients  were  used  to  analyze  the  data. 

Issue  of  Validity  Addressed:  "There  are  several  weaknesses 
in  the  proposed  design  that  need  mentioning.  First,  we  are 
relying  on  people's  memories  for  data  on  the  system- 
development  process ....  A  second  weakness  in  our  design  is 
the  fact  that  we  have  to  rely  on  the  perceptions  of  people 
instead  of  making  direct  observations....  Third,  our 
results  are  limited  to  the  particular  roles  in  the 
development  process  that  we  studied.  Application  to  systems 
development  in  general  must  be  made  with  caution,  since  we 
only  have  one  informant  per  group." 
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Tvp«  of  Validity  Clainwd:  None. 

Source  of  Val.^<^^*‘-Y  claimed:  Not  discussed. 
Type  of  Data  Recorded:  Ordinal. 
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King,  W.  R.  and  J.  I.  Rodriguez,  "Evaluating  Management 
Information  Systems, "  Management  Information  System 
Quarterly,  Vol .  2,  No.  3,  (September  1978),  pp.  43-51. 


Description  of  System  Studied:  A  system  designed  to  aid 
managers  in  understanding  and  resolving  competitor-related 
strategic  issues.  It  was  a  system  that  combined  an 
intelligent  management  information  system  and  a  management 
decision  support  system. 

Data  Collection  Technique (s)  Used:  Experiment  using  a 
questionnaire . 

Characteristics  of  Measuring  Instruments  Used:  Experiment 
conducted  in  a  simulated  business  environment.  True 
experiment,  pre/post  test  and  post  test  only,  control  group 
design.  Questionnaire  was  developed  by  Schultz  and  Slevin, 
employing  ordinal  scale . 

Objective (s)  of  Measurement  Process:  The  questionnaire 
assessed  attitudes  and  value  perceptions. 

Participants  in  Measurement  Process :  Forty-five  experienced 
manager-users,  who  were  enrolled  in  part-time  MBA  program. 

Distribution  of  heasurinq  Instruments :  Questionnaire  given 
to  students  participating  in  MBA  program. 

Corrective  Action  taken  to  Account  for  Won-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Methods  used  to  analysis  data 
were  Schultz-Slevin  "dependent  variables",  Kendall  Rank 
Correlation  test,  and  possible  regression. 

Issue  of  Validity  Addressed:  "Clearly,  behavior  in  the 
simulated  business  environment  in  which  the  evaluation  was 
performed  may  not  be  closely  related  to  behavior  in  the  real 
world  which  may  raise  questions  as  to  the  external  validity 
of  the  experiment . " 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Not  discussed. 
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Type  of  Data  Recorded:  Ordinal. 


Secondary  Source (s)  Cited:  Schultz  and  Slevin  (1975)  is 
where  the  development  of  the  original  instrument  was 
presented,  however,  its  validity  was  not  clearly 
estcJDlished. 
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Elchesen,  D.  R. ,  "Cost-Effectiveness  Comparison  of  Manual 
and  On-line  Retrospective  Bibliographic  Searching, "  Journal 
of  American  Society  for  Information  Science,  Vol .  29,  No.  2, 
(March  1978),  pp.  56-66. 


Description  of  System  Studied:  Manual  and  on-line 
bibliographic  information.  Forty  search  queries  were 
processed  against  seven  abstracting-indexing  publications 
and  the  corresponding  SDC/ORBIT  data  base . 

Data  Collection  Technique (s)  Used:  Experiment  relying  on 
observation  of  participants. 

Characteristics  of  Measuring  Instruments  Used:  Participants 
measured  variables  of  interest  and  recorded  results  on  logs . 
Participants  assigned  values  to  the  effectiveness  of  the 
search  results. 

Objective (s)  of  Measurement  Process:  Measures  of 
performance  were  implied.  Data  collected  were  not 
specifically  identified  as  measures  of  attitude,  opinion, 
behavior,  or  etc.. 

Participants  in  Meas”**^****"^.  Process:  Eight  of  thirteen 
members  of  the  Lawrence  Livermore  LzU^oratory  Research 
Information  Group.  The  group  was  charged  with  providing 
bibliographic  information  service  to  the  Laboratory' s 
scientific  and  engineering  staff.  All  were  experienced  with 
manual  zmd  on-line  bibliographic  searching. 

Distribution  of  Measuring  inatiiinients :  Not  discussed. 

Corractiv  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non- 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Descriptive  statistics  were 
used  to  analyze  the  data. 

Issue  of  Validity  Addressed:  Researchers  generalize  results 
of  study. 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative  and  metric. 
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Robey,  D.  and  R.  L.  Zeller,  "Factors  Affecting  the  Success 
and  Failure  of  an  Information  System  for  Product  Quality, " 
Interfaces,  Vol.  8,  No.  3,  (February  1978),  pp.  70-75. 


Description  of  System  Studied:  Quality  information  system 
used  by  the  product  quality  section  of  two  neighboring 
plants  of  a  division  of  a  very  large  U.S.  corporation.  The 
divisions  manufactured  a  wide  variety  of  highly  teclxnical 
equipment  of  the  health  care  industry. 

Data  Collection  Technique (s>  Deed:  Interview,  supplemental 
questionnaire,  analysis  of  existing  records  and  documents, 
and  interviews . 

Characteristics  of  Measuring  Instrxunents  Used;  No 

information  regarding  the  conduct  of  the  interview.  Company 
documents  were  used  during  the  analysis  of  existing 
documents  and  records .  Questionnaire  was  developed  by 
Schultz  and  Slevin. 

Objective (s)  of  Measurement  Process:  Interviews  gathered 
general  information  about  the  implementation  process  in  each 
department .  The  interview  and  company  documents  were  used 
to  identify  some  differences  in  structure  between  the 
departments .  Questionnaires  measured  attitudes .  "Responses 
show  the  user's  perceptions  of  how  favorable  the  system  is 
for  each  area  of  concern." 

Participants  in  Measurment  Process;  Eleven  users . 
Distribution  of  Heasurino  •  Not  discussed. 

Corrsctiv  Action  takan  to  Account  for  Non-participants; 

Not  discussed. 

Identification  of  Diffarancss  batwn  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Mann-Whitney  U  test  was  used 
to  analyze  the  data. 

Issue  of  Validity  Addressed:  "While  we  cannot  say  that 
attitudes  were  the  cause  of  behavior  in  this  case,  certain 
attitudes,  and  not  others,  are  related  to  behavior.”  "The 
essential  advantage  of  our  research  design  is  its  ability  to 
compare  attitudes,  organization  structure  and  implementation 
process  while  holding  the  MIS  features,  production 
technology  and  corporate  policy  constant.  By  controlling 
the  effects  of  these  other  factors,  the  effects  of  the 
behavioral  and  organizational  factors  can  be  more  readily 


identified . ” 


Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal. 

Secondary  Source (s)  Cited:  Schultz  and  Slevin  (1975)  is 
where  the  development  of  the  original  instrument  was 
presented,  however,  its  validity  was  not  clearly 
established. 
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Lucas,.  H.  C...  Jr.,  "Unsuccessful  Implementation;  The  Case 
of  a  Computer-Based  Order  Entry  System, "  Decision  Sciences, 
Vol .  9,  No.  1,  (January  1978),  pp.  68-79. 


Description  of  System  Studied:  Optical  character 
recognition  system  and  keypunch  system  for  order  entry 
procedures  conducted  in  two  divisions  of  a  large  West  Coas 
apparel  manufacturing  company. 

Data  Collection  Technique fs)  Used:  Experiment  and 
questionnaire 

Characteristics  of  Measuring  Instruments  Used:  Experiment 
employed  a  control  group  and  excunined  the  implementation  of 
a  system  in  a  longitudinal  field  study.  The  questionnaire 
used  scaled  varieibles  form  1  to  7.  M/N,iified  version  of  the 
JDI  work  scale  was  used  to  measu'»*e  satisfaction. 


Objective (s)  of  Measureme^A  Process:  Questionnaire 
success  of  implementation,  costs,  and  benefits. 

Participants  in  Measurement  Process:  Sales  representatives. 

Distribution  of  Measuring  Instruments:  Distribution  to  the 
experimental  group  (approximately  120  sales  representative) 
is  unknown.  Questionnaires  were  mailed  to  the  control  group 
of  70  randomly  selected  sales  representatives. 

Approximately  three  months  later  a  follow-up  questionnaire 
was  mailed.  Response  to  the  second  questionnaire  was  56  for 
the  experimental  group  and  58  for  the  control  group. 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 


Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Dynamic  and  cross-lagged 
correlational  analysis  was  employed. 

Issue  of  Validity  Addressed:  "The  c[uasi-experimental 
research  design  and  these  statistical  techniques  provided 
increased  confidence  eibout  causal  inferences  concerning  the 
relationship  between  the  benefits  and  costs  of  the  new 
system  and  successful  implementation." 

Type  of  Validity  Claimed:  Internal  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 
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Tvpa  of  Data  Recorded:  Ordinal. 

Socondarv  Source (s)  Smith,  Kendall,  and  Hulin  (1969) 

present  original  instrximent  and  established  its  validity . 
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Neumann,  S.  and  E.  Segev,  ”A  Case  Study  of  User  Evaluation 
of  Information  Characteristics  for  Systems  Improvement, " 
Information  &  Management,  Vol .  2,  (1979),  pp.  271-278. 


Description  of  Svy^**™  studied:  A  computerized  information 
system  which  served  148  branches  of  one  of  the  three  leading 
banks  in  Israel. 

Data  Collection  Technique (e)  Used:  Interview  and 
questionnaire  were  administered  jointly. 

Characteristics  of  Measuring  Instruments  Used:  The 

interview  was  unstructured  and  scheduled.  A  1  to  5  scale 
was  used  in  the  questionnaire. 

Objective (s)  of  Measurement  Process:  The  data  were  measures 
of  the  respondents  perceptions. 

Participants  in  Measurement  Process:  Branch  managers . 

of  Measuring  instrum^^tw  Interviews  conducted 
in  person.  Questionnaires  were  provided  by  hand  to  130 
branch  managers .  Eighty-one  sets  of  interviews  and 
questionnaires  were  properly  completed  and  returned  by  mail . 

Corrective  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Guttman' s  correlation 
coefficients  of  weak  monotonicity. 

Issue  of  Validity  Addressed:  "We  believe  the  findings  may 
have  implications  for  other  organizations....  The  findings 
have  no  implications  on  the  a  priori  users'  definitions  of 
their  needs,  only  on  perceptions  of  existing  systems.... 
Further  research  in  other  organizations, ...  is  necessary  to 
validate  whether  the  findings  of  this  study  are  unique  or 
typical  of  other  organizations." 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal. 
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Robey,  D.,  "User  Attitudes  and  Management  Information  System 
Use,"  Academy  of  Management  Journal,  Vol .  22,  No.  3, 
(September  1979),  pp.  527-538. 


Description  of  System  Studied:  Computer  based  system  to 
record,  update,  and  maintain  information  pertaining  to 
customer  sales  accounts .  The  system  was  used  by  sales 
members  of  a  large  industrial  products  manufacturer. 

Data  Collection  Technique (s)  Osed:  Analysis  of  existing 
documents  and  records  and  questionnaire. 

Characteristics  of  Measuring  Instruments  Used:  Two  kinds  of 
account  records  were  the  existing  docxaments  and  reports 
used.  A  Schultz  and  Slevin  type  of  questionnaire  was  used. 
Both  a  five  point  and  a  ten  point  scale  was  used  in  the 
questionnaire . 

Objective (s)  of  Measurement  Process:  Account  records  were 
analyzed  to  measure  system  usage.  The  cjuestionnaire 
measured  users'  attitudes. 

Participants  in  Measurement  Process;  Members  of  the  sales 
force . 

Distribution  of  Measuring  Instruments:  Corporate  mail  was 
used  for  distribution  and  a  100%  return  rate  was  achieved. 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  required. 

Identification  of  Differences  between  Participants  emd  Non¬ 
participants  :  No  non-participants. 

Method  Used  to  Analyse  Data:  Cronbach' s  coefficient  alpha, 
Spearman  rank  correlations,  and  Pearson  product -moment 
correlations . 

Issue  of  Validity  Addressed:  "In  returning  to  the  more 
direct  implications  of  this  research,  it  is  important  to 
note  its  limitations.  First,  although  the  Schultz  and 
Slevin  instrument  appears  to  be  superior  to  single-item 
scales  used  elsewhere,  it  has  not  received  extensive 
validation. . . .  Second,  a  word  of  caution  must  be  extended 
to  those  interpreting  these  results  and  the  results  of  any 
cross-sectional  study.  Strong  positive  relationships  have 
been  demonstrated  between  specific  user  attitudes  and  actual 
use  of  an  MIS.  It  does  not  necessarily  follow,  however, 
that  attitudes  cause  behavior." 


Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal,  frequency,  and  interval. 

Secondary  Source (a)  Cited:  Schultz  and  Slevin  (1975)  is 
where  the  development  of  the  original  instrument  was 
presented,  however,  its  validity  was  not  clearly 
estcQ^lished. 
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Mansour,  A.  H.  and  H.  J.  Watson,  "The  Determinants  of 
Computer  Based  Information  System  Performance, "  Academy  of 
Management  Journal,  Vol .  23,  No.  3,  (September  1980),  pp. 
521-533. 


Description  of  System  Studied:  Computer  based  information 
systems  in  engineering,  finance,  manufacturing, 
retail/wholesale,  service,  and  government. 

Data  Collection  Technique (s)  Used:  Two  questionnaires. 

Characteristics  of  Measuring  Instruments  Used:  User 
questionnaire  used  a  five  point  scale.  Expert  questionnaire 
was  used  for  design  of  experiment. 

Objective (s)  of  Measurement  Process:  Questionnaire 
completed  by  users  obtained  data  on  behavioral,  structural, 
environmental,  and  performance  variables.  Questionnaires 
completed  by  experts  identified  study  variad^les  and  assigned 
weights  to  the  variadales . 

Participants  in  Measurement  Process:  Users  and  expert  panel 
(three  practitioners  and  three  academicians  in  the  CBIS 
field) . 

Distribution  of  Measuring  >  Unknown  quantity  of 

user  questionnaires  were  mailed.  A  total  of  108 
questionnaires  were  returned.  Distribution  of  expert 
questionnaire  is  unknown. 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Step-wise  regression. 

Issue  of  Validity  Addressed:  Author  implies  internal 
validity  and  external  validity. 

Type  of  Validity  Claimed:  Internal  and  external  (implied) . 
Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal,  qualitative,  and  interval. 


119 


Ball,  L.  D.,  "Increasing  the  Manager's  Productivity  Through 
the  Information  System,"  Public  Productivity  Review,  Vol .  4, 
No.  1,  (March  1980),  pp.  51-62. 


laentirxcation  or  Dxrfernces  periren  Particxpants  and  Non- 
participantg ;  States  possible  reasons  some  managers  were 
non-users  of  the  system,  thus  making  them  non-participants 
in  the  study. 

Method  Usad  to  Analyze  Data:  Descriptive  statistics. 

Issue  of  Validity  Addressed:  Not  discussed. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative,  Metric,  Interval,  and 
Frequency . 


Edelman,  F.,  "Managers,  Computer  Systems,  and  Productivity," 
Management  Information  System  Quarterly,  Vol .  5,  No.  3, 
(September  1981),  pp .  1-19. 


Description  of  Synt-^in  Studied:  Industrial  Relations 
Information  System  at  RCA,  used  to  provide  MIS  support 
relating  to  human  resource  mcinagement .  The  system  included 
data  base,  interactive,  office  support,  and  management 
support  technologies . 

Data  Collection  Technique (s)  Used:  Controlled  experiment 
and  apparently  analysis  of  existing  records  and  docviments . 

Characteristics  of  Measuring  Instruments  Used:  Experiment 
employed  a  target  and  control  group.  Apparently  archival 
records  pertaining  to  personnel  costs  were  used  during  the 
experiment . 

Objective (s)  of  Measurement  Process:  Apparently  the  data 
collected  measured  costs. 

Participants  in  Measurement  Process:  Fourteen  of  RCA' s 
eighteen  operating  units . 

Distribution  of  Measuring  Instruments:  Not  discussed. 

Corrective  Action  talctm  to  Account  for  Non~participants : 
None  required. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Clearly  identified. 

Method  Used  to  Analyze  Data;  Descriptive  statistics. 

Issue  of  Validity  Addressed:  Only  the  validity  of  the 
experimental  design  used. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval  and  frequency. 
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Altmeyer,  A.  S.  and  L.  S.  Bozeman^  "Productivity  via 
Computer  Technology,"  Public  Productivity  Review,  Vol .  5, 
No.  4,  (December  1981),  pp.  321-334. 


Description  of  System  Studied:  Welfare  Management  System, 
an  information  system  designed  to  receive  and  process 
information  on  clients  who  have  applied  for  or  have  been 
determined  eligible  for  public  assistance.  Medicaid 
Management  Information  System,  an  information  system 
designed  to  improve  the  availability  of  health  care  services 
as  well  as  administrative  controls  over  rising  costs. 

Data  Collection  Tschnioue ( s )  Used:  Interviews  and  two 
questionnaires.  Interviews  and  first  questionnaire  was  used 
to  design  experiment. 

Characteristica  of  Measuring  Inatrunants  Usad:  A  series  of 
open-ended,  unstructured  field  interviews.  Semi-structured 
six-question  telephone  questionnaire,  which  asked  open-ended 
questions  regarding  performance  benefits.  Survey 
questionnaire  was  self-administered,  containing  102 
questions.  The  questionnaire  required  the  responded  to 
select  "agree,  do  not  know,  or  disagree"  for  statements.  It 
was  reviewed  by  a  panel  of  experts,  before  administration. 

Obiaetiva(a)  of  Maaaurmant  Proeaaa:  "Responses  (of  survey 
questionnaires)  showed  perceptional  judgements  of  middle 
management...."  Author  presents  findings  gathered  from 
survey  data  regarding  managerial  aspects;  productivity; 
organizational  effectiveness;  and  several  user  attitudes. 

Participants  in  Measurement  Process;  Interviews  were 
conducted  with  personnel  serving  in  several  levels  of  county 
social  services  departments .  Additional  interview  were  held 
with  a  number  of  county  commissioners  as  well  as  officials 
at  the  state  office  in  Albany.  The  telephone  q[uestionnaire 
was  administered  to  county  staff  developmental  personnel . 
Survey  questionnaire  was  administered  to  various  levels  of 
managers . 

Distribution  of  Measuring  •  interviews  were 

conducted  in  person,  no  response  rate  was  provided. 

Telephone  questionnaire  was  administered  over  the  telephone, 
with  44  of  56  staff  development  coordinators  responding. 
Survey  questionnaires  were  mailed  to  492  managers,  with  a 
response  rate  of  67%. 

Corrective  Action  taken  to  Accovint  for  Non-participamts : 

Not  discussed. 
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Identification  of  Differences  between  Participants  and  Non- 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data :  Descriptive  statistics. 

Issue  of  Validity  Addressed:  Results  of  study  were 
generalized . 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative. 
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Ginzberg,  M.  J.,  "Early  Diagnosis  of  MIS  Implementation 
Failure:  Promising  Results  and  Unanswered  Questions." 

Management  Science,  Vol .  27,  No.  4,  (April  1981),  pp.  459- 


478. 


Description  of  System  Studied:  On-Line  Portfolio  Management 
system,  an  information  system  used  to  support  portfolio 
managers  in  their  job  of  managing  asset  portfolios 
consisting  primarily  of  common  stock,  debt  instrxaments,  and 
cash.  The  system  was  operated  in  the  trust  department  of  a 
large  U . S .  bank . 

Data  Collection  Technique (s)  Used:  Interview,  two 
questionnaires,  and  analysis  of  existing  documents  and 
records . 

Characteristics  af  Measuring  lna»i-mnents  Used:  Interview 
used  to  devexo  .1  pre- implementation  questionnaire.  Pretest 
of  the  pre  ^  plementation  questionnaire  was  conducted.  Pre- 
implementa.ti.on  questionnaire  consisted  of  open-ended 
questiotis.  Post- implementation  questionnaire  used  closed- 
ended  qpaestions  and  scales  questions.  Post-implementation 
analysis  of  existing  documents  and  records  was  collected  by 
systems  software  to  measure  system  individual  system  usage. 

Object iv (a)  of  Meaauraznant  Process:  Author  stated  that 
three  major  types  of  data  were  collected:  1)  users'  pre¬ 
implementation  expectations,  2)  expert  group  pre¬ 
implementation  expectations  euid  3)  users'  post- 
implementation  assessment  of  success.  Post -implement at ion 
data  were  collected  to  measure  attitudes  and  behavior. 
Attitudes  were  measured  by  questionnaire  and  behavior  by 
archival  records . 

Participants  in  Msasurmut  Procsss:  Interviews  to  develop 
pre-implementation  questionnaire  were  conducted  with  a 
sample  of  portfolio  managers  (15%) ,  system  developers  (both 
internal  and  external) ,  trust  department  management,  and 
other  trust  department  personnel.  Pre- implement at ion 
questionnaire  was  administered  to  portfolio  managers  and 
others.  Post-implementation  questionnaire  was  administered 
to  portfolio  managers  who  had  responded  to  the  pre¬ 
implementation  questionnaire. 
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Distribution  of  Measuring  Inst.r\iments :  Interviews  appear  to 
have  been  conducted  in  person.  Pre-implementation  interview 
was  mailed  to  44  portfolio  managers  and  21  others,  with  86% 
and  82%  response  rates  achieved  respectively.  Post¬ 
implementation  interview  was  mailed  to  the  portfolio 
managers  and  a  82%  response  rate  was  achieved. 

Corrective  Action  taJcen  to  Account  for  Non-participants : 

Follow-up  message. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Differences  were  discussed.  "Analysis  of  the 
available  data  shows  no  significant,  nor  even  sizable, 
difference  between  respondents  and  non-respondents  on  pre¬ 
implementation  demographic  or  attitudinal  measures." 

Method  Used  to  Analyze  Data:  Pre- implement at ion  expectation 
data  were  analyzed  using  McQuitty' s  Elementary  Linkage 
Analysis.  Post-implementation  data  were  analyzed  using 
Kendall's  tau  correlations. 

Issue  of  Validity  Addressed:  "While  there  is  no  a  priori 
reason  to  believe  that  the  specific  characteristics  of  this 
situation  limit  the  generalized>ility  of  the  results,  there 
is  also  no  evidence  to  support  such  generalizations . " 

Type  of  Validity  Claimed:  Limited  external  (implied) . 

Source  of  Validity  Claimed :  None . 

Type  of  Data  Recorded;  Ordinal  and  frequency. 
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Cron,  W.  L.  and  M.  G.  Sobol,  "The  Relationship  Between 
Computerization  and  Performance;  A  Strategy  for  Maximizing 
the  Economic  Benefits  of  Computerization, "  Information  & 
Management,  Vol .  6,  No.  3,  (June  1983),  pp.  171-181. 


Not  discussed. 


Identification  of  Digggrencqs  between  Participants  and  Non¬ 
participants  :  Major  non-participants  identified  and 
differences  discussed. 


Method  Uaed  to  Analyze  Data:  Descriptive  statistics,  Chi- 
sc3paare  analysis,  and  analysis  of  variance. 


Issue  of  Validity  Addressed:  "Though  it  is  always  difficult 
to  establish  causality  from  correlation  data  conclusively,  a 
number  of  significant  relationships  were  found. . . .  While 
recognizing  that  it  is  important  to  replicate  this  study 
using  different  kinds  of  firms  and  industries,  results  of 
this  study  indicate  that  computerization  is  positively 
related  to  the  overall  performance  of  medium  and  larger  size 
firms . " 


Type  of  Validity  Claimed:  External  (implied) . 
Source  of  Validity  Claimed:  Not  discussed. 


Type  of  Data  Recorded:  Qualitative,  Interval,  and  possible 
others . 
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Bruwer,  P  .J  .S.,  "A  Descriptive  Model  of  Success  for 
Computer-Based  Information  Systems, "  Information  & 
Management,  Vol .  1,  No.  2,  (July  1984),  pp.  63-67. 


Deacription  of  System  Studied:  140  computerized  systems, 
used  by  about  1200  clerical  staff  and  114  managers,  in  a 
very  large  organization. 

Data  Collection  Technique (a)  Uaed:  Questionnaire. 

Characteristics  of  Measuring  Instruments  Used:  Most  of  the 
questions  used  a  7  point  scale. 

Obiective(s)  of  Measurement  Process:  Questionnaire 
quantified  variaibles  in  a  model,  and  these  variadsles  were 
used  to  measure  attitude,  quality,  success  of  system,  and 
management  involvement . 

Participants  in  Measurement  Process :  Managers. 

Diatr-ibution  of  Measuring  lnstru««»"t« •  Not  discussed.  Full 
participation  of  114  managers. 

Corrective  Action  taken  to  Accotint  for  Non-participants: 

Not  required,  full  participation. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  required,  full  participation. 

Method  Used  to  Analyse  Data:  Using  BMDP  statistical 
computer,  correlation  and  regression  methods  were  used. 

Issue  of  Validity  Addressed:  Generalized  the  results  of  his 
study . 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal. 
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Decker,  J.  E.  and  J.  P.  Plvunlee,  "Microcomputers,  Manpower 
and  Performance  in  a  Public  Utility, "  Public  Productivity 
Review,  Vol .  9,  No.  2-3,  (Siammer/Fall  1985),  pp.  213-225. 


Description  of  System  Studied:  A  work  force  management 
system,  operated  by  the  Jacksonville  Electric  Authority. 

Data  Collection  Technique (s)  Used:  Analysis  of  existing 
documents  and  reports . 

Characteristics  of  Measuring  Instruments  Used:  Data  base 
information  was  analyzed  as  existing  documents  and  reports. 

Objective (s)  of  Measurement  Process:  Measurement  of 
performance . 

Participants  in  Measurement  Process :  Case  study  of  one 
utility  company's  experience. 

Distribution  of  Measuring  •  Not  discussed. 

Corrective  Action  taken  to  Account  for  Non-part i c ipemt s : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Descriptive  statistics. 

Issue  of  Validity  Addressed;  Generalized  results.  Implied 
external  validity.  Did  not  address  internal  validity. 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type 


of  Data  Recorded:  Metric,  Interval,  and  Frequency. 


Englander,  V.  and  F.  Englander,  "Cost-Effectiveness  of 
Computer-Based  Welfare  Fraud  Detection  in  New  Jersey, " 
Public  Productivity  Review,  Vol .  9,  No.  2-3,  (Svimmer/Fall 
1985),  pp.  271-282. 


Description  of  Svetexa  Studied:  Software  program  which  was 
used  to  detect  the  fraudulent  receipt  of  public  assistance 
benefits  by  participants  in  the  Aid  to  Feunilies  with 
Dependent  Children,  the  Food  Staunps,  and  the  Medicaid 
prograuns  of  New  Jersey. 

Data  Collection  Technique (s)  Used:  Analysis  of  existing 
records  and  documents . 

Characteristics  of  Measuring  Instruments  Used:  Computer 
records  were  analyzed  quarterly  to  match  SSN  of  welfare 
recipients  against  employed  taxpayers . 

Objective (s)  of  Measurement  Process:  Data  were  used  to 
measure  cost  savings .  The  value  was  determined  by  keeping  a 
running  total  of  reductions  in  benefits  directly  from 
closures  and  grant  reductions  detected  by  computer  matches . 

Participants  in  Measurement  Process:  New  Jersey  Division  of 
Pxiblic  Welfare. 

Distribution  of  Measuring  •  Not  discussed. 

Corrective  Action  taken  to  Account  for  Non-participants: 
Assumed  full  participation. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Assumed  full  participation. 

Method  Used  to  Analyse  Data:  Regression  analysis. 

Issue  of  Validity  Addressed:  Addressed  possible  problems 
with  the  internal  validity.  External  validity  was  not 
discussed. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval. 
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Sullivan,  R.  S.  and  S.  C.  Secrest,  "A  Simple  Optimization 
DSS  for  Production  Planning  at  Dairyman's  Cooperative 
Creaunery  Association,"  Interfaces,  Vol .  15,  No.  5, 
(September-October  1985),  pp.  46-53. 


of  System  Studied:  A  small  optimization-based 
decision  support  system  that  was  used  to  perform  milk  flow 
analysis  at  Dairyman's  Cooperative  Creamery  Association. 

Data  Collection  Technique (e)  Used:  Analysis  of  existing 
docviments  and  reports . 

Characteristics  of  Measuring  Inatnamenta  Psed:  Reports 
generated  by  the  milk  flow  analysis  program  were  the 
existing  documents  and  reports  analyzed. 

Objective fs)  of  Measurement  Process:  Reports  generated  by 
the  milk  flow  program  were  analyzed  to  measure  system 
performance. 

Participants  in  Measurement  Process:  Several  plant 
supervisors  who  had  been  involved  with  production  planning 
for  more  than  20  years. 

Distribution  of  Measuring  Instruments:  Not  discussed. 
Response  rate  was  not  discussed. 

Corrective  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Not  discussed. 

Issue  of  Validity  Addressed:  Not  discussed. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval  and  metric. 
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Lincoln,  T.,  "Do  Computer  Systems  Really  Pay-off?," 
Information  &  Management,  Vol .  11,  No.  1,  (August  1986),  pp. 
25-34. 


Description  of  System  Studied:  Systems  consisting  of  167 
individual  applications,  ran  by  20  companies.  These 
applications  reside  on  integrated  systems,  stand-alone 
systems,  and  general  facility/service  systems. 

Data  Collection  Technique (s>  Used:  Analysis  of  existing 
documents  and  records,  interview,  and  questionnaire. 

Characteristics  of  Measuring  Instruments  Used:  Existing 
documents  and  records  were  archival  data  such  as  accounting 
reports .  Each  interview  was  customized  and  incorporates  a 
questionnaire.  The  participant  reviewed  and  signed-off  on 
the  results  of  the  interview. 

Objective (s)  of  Measurement  Process:  Analyses  of  existing 
documents  and  records  were  a  measure  of  costs .  Interviews 
and  questionnaires  were  used  to  identify  (measure)  benefits. 

Participants  in  Measurement  Process:  Users. 

Distribution  of  Measuring  TnatyiimantM-  Distribution  of 
existing  documentation  was  not  discussed.  Interview  and 
questionnaire  appears  to  have  been  administered  in  person. 
Response  rate  was  not  discussed. 

Corrective  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Descriptive  statistics  and 
financial  analysis  techniques  (break-even  point,  mean 
benefit/cost  ratio,  and  internal  rates  of  return. 

Issue  of  Validity  Addressed:  The  author  addressed 
unreliability  of  user  interviews. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval,  Frequency,  and  possible 
others . 
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Franz,  C.R.  and  D.  Robey,  ”  Organizational  Context,  User 
Involvement,  and  the  Usefulness  of  Information  Systems," 
Decision  Sciences,  Vol.  17,  No.  3,  (Svimmer  1986),  pp.  329- 
355. 


Description  of  System  Studied:  Information  systems  used  by 
34  organizations.  The  types  and  modes  of  interaction  varied 
greatly  between  systems. 

Data  Collection  Technicrue  (s>  Used:  Interview  and 
questionnaire . 

Characteristics  of  Measuring  Instruments  Used: 

Characteristics  of  the  interview  were  not  discussed.  A 
cross-sectional  field  survey.  Questionnaire  was  Likert 
type. 

Objective (s)  of  Measurement  Process:  The  interviews  yielded 
a  preliminary  view  of  its  computer  and  information  system. 
Questionnaire  was  used  to  measure  user's  involvement  and 
perceived  usefulness  of  system. 

Participants  in  Measurement  Process:  Initial  interviews 
were  with  senior  MIS  managers.  Questionnaires  were 
completed  by  user-managers . 

Distribution  of  Measuring  •  of  the  49  managers 

contacted  for  interviews,  44  agreed  to  cooperate. 
Distribution  of  the  qpiestionnaire  was  not  discussed.  A 
total  of  188  user-managers  in  34  organizations  returned  the 
questionnaire.  The  response  rate  was  not  discussed. 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Canonical  correlation  models 
were  constructed  using  Statistical  Package  for  Social 
Sciences,  release  7.0  to  analyze  data. 

Issue  of  Validity  Addressed:  Authors  stated  that 
standardized  instruments  for  measuring  user  involvement  and 
system  usefulness  were  unavailable  at  the  start  of  the 
study.  They  had  to  develop  and  validate  an  approach  to 
measure  such  variables.  They  clearly  described  how  the 
questionnaire  instrument  was  developed  and  tested.  Imply 
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that  the  external  validity  of  their  instriunents  have  not 
been  este±»lished. 

Type  of  Validity  Claimed:  Internal  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Ordinal. 
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Kapsales,  P.,  "Before  and  After  WP :  An  Office  Automation 
Productivity  Study,"  Journal  of  Systems  Management,  Vol .  37, 
No.  4,  (April  1986),  pp.  7-9. 


Not  discussed. 


Identification  of  Differancas  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyse  Data:  Descriptive  statistics. 

laeue  of  Validity  Addressed:  Presented  as  one-shot  case 
study,  but  provides  a  few  generalizations  for  other 
organizations . 

Type  of  Validity  Claimed:  External  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval. 
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Snitkin,  S.R.  and  W.  R.  King,  "Determinants  of  the 
Effectiveness  of  Personal  Decision  Support  Systems, " 
Information  &  Management,  Vol.  10,  No.  2,  (January  1986), 
pp.  83-89. 


Description  of  System  Studied:  Personal  decision  support 
systems . 

Data  Collection  Technique (s)  Used:  Questionnaire. 

Characteristics  of  Measuring  Instruments  Used: 

Questionnaire  used  a  five  point  scale  and  open-ended 
questions . 

Objective (s)  of  Measurement  Process:  Questionnaire  measured 
user  perception  of  system  effectiveness  and  the  open-ended 
question  allowed  users  to  describe  the  most  important 
applications  pertaining  to  effectiveness . 

Participants  in  Measurenent  Process:  Users  of  personal 
decision  support  systems,  purchased  from  one  vendor. 

Distribution  of  Measuring  Instrumants :  Questionnaires  were 
mailed  to  310  users.  A  response  rate  of  18%  was  achieved. 

Corrsctivs  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Diffarancas  batwean  Participants  and  Non¬ 
participants  :  "The  respondent  anonymity  requested  by  the 
computer  deal  precluded  an  auialysis  of  non -respondents .  " 

Method  Used  to  Analyse  Data:  Chi-square  analysis. 

Issue  of  Validity  Addressed:  Not  addressed. 

Type  of  Validity  Claimed:  Not  discussed. 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative,  ordinal,  and  possible 
others . 
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Aldag,  R.  J.  and  D.  J.  Power,  "An  Empirical  Assessment  of 
Computer  Assisted  Decision  Analysis, "  Decision  Sciences, 
Vol.  17,  No.  14,  (Fall  1986),  pp.  572-588. 


Description  of  System  Studied:  DECAID,  a  first-generation 
computerized  decision-analysis  program.  The  program  helped 
the  user  conduct  a  structured  decision  analysis,  using 
question-and-response  dialogues. 

Data  Collection  Technique (s)  Used:  Experiment  and 
questionnaire . 

Characteristics  of  Measuring  Instruments  Used:  Experiment 
was  conducted  in  the  laboratory.  Questionnaire  was  Likert 
typer  using  five  and  seven  point  scales. 

Objective (s)  of  Measurement  Process:  Experiment  was  used  to 
introduce  post-test  control  group  into  design.  The 
questionnaire  measured  attitudes. 

Participants  in  Measurement  Process:  Business  students  in 
undergraduate  and  graduate  level  normative  decision-making 
courses . 

Distribution  of  Measuring  »  Assumed  to  have  been 

distributed  in  class . 

Corrective  Action  taken  to  Account  for  Won-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Variety  of  methods  used  to 
analyze  correlation  (Principal  component  factor  analysis 
with  varimax  rotation,  analysis  of  inter  rater  reliability, 
Dimensional  interactive  cognitive  complexity,  tolerance  for 
ambiguity, . . .  and  one-way  analysis  of  variance.) 

Issue  of  Validity  Addressed:  Internal  validity  was  strongly 
implied  and  authors  directly  addressed  face  validity  of 
program.  "In  conclusion,  computerized  decision-analysis 
aids  may  serve  many  useful  functions.  However,  the  only 
benefits  clearly  evident  at  present  are  that  the  programs 
offer  students  experience  with  a  new  decision  technology  and 
provide  a  structured  experience  in  decision  making.  These 
programs  seem  to  have  high  face  validity  and  may  result  in 
positive  user  affect  and  enhanced  confidence."  External 
validity  was  addressed,  "Clearly,  additional  studies  are 
needed  before  confident  prescriptions  can  be  made  concerning 
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the  applicability  and  benefits  of  computerized  decision  aids 
for  management  decision  making." 

Type  of  Validity  Claimed:  Internal. 

Source  of  Validity  Claimed:  Analysis  of  instrument 
measures . 

Type  of  Data  Recorded:  Ordinal. 
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Baroudi,  J.  J.,  M.  H.  Olson  and  B.  Ives,  "An  Empirical  Study 
of  the  Impact  of  User  Involvement  on  System  Usage  and 
Information  Satisfaction,"  Communications  of  the  ACM,  Vol . 
29,  No.  3,  (March  1986),  pp.  232-238. 


Description  of  System  Studied:  A  variety  of  information 
systems  in  over  200  large  manufacturing  firms. 

Data  Collection  Technique (s>  Used:  Questionnaire. 

Characteristics  of  Measuring  s  Used: 

Questionnaire  developed  and  partially  validated  by  Bailey 
and  Pearson. 

Objective (s)  of  Measurement  Process:  Measures  of 
information  satisfaction,  system  usage,  and  user 
involvement . 

Participants  in  Measurement  Process :  Production  managers 
from  over  200  large  manufacturing  firms. 

Distribution  of  Measuring  Tnatinmants :  A  total  of  800 
questionnaires  were  mailed,  with  200  subjects  responding. 

Corrective  Action  taken  to  Account  for  Non-participants: 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Briefly  mentioned  that  old  mailing  lists  were 
used,  trying  to  account  for  non-response. 

Method  Used  to  Analyse  Data:  Zero  order  correlations  were 
calculated,  a  correlation  matrix  developed,  and  path 
coefficients  calculated. 

Issue  of  Validity  Addressed:  "Data  generated  in  this  study 
were  also  used  to  further  validate  and  modify  this 
instrument . "  "Even  though  the  data  were  collected  in  a 
cross-sectional  survey,  attention  was  paid  to  the  causal 
ordering  of  the  variables."  "Generalized>ility  across 
organizations  and  systems  was  enhanced  by  including  over  200 
users  in  the  saune  function  from  many  different 
organizations."  "Perceptual  rather  than  objective  measures 
of  system  usage  were  utilized."  Authors  clearly  addressed 
internal  and  external  concerns  in  their  conclusions. 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  Analysis  of  instrument 
measures . 
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Type  of  Data  Recorded:  Ordinal. 


Secondary  Source (s)  Cited:  Bailey  and  Pearson  (1983)  is 
where  the  development  of  the  original  instrument  was 
presented.  Its  validity  was  discussed,  claiming  content, 
construct,  and  external  validity.  The  source  of  the  claim 
was  an  analysis  of  the  instruments  measurements. 
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Franz,  C.  R.,  D.  Robey  and  R.  R.  Koeblitz,  K^.^ponse  to 

an  Online  Information  System:  A  Field  Experiment, " 
Management  Information  Systems  Quarterly,  (March  1906),  pp. 
29-42 . 


Description  of  System  Studied:  Patient  Care  System, 
developed  jointly  by  the  Duke  University  Medical  Center  and 
IBM,  as  operated  in  a  large  hospital  with  over  400  beds, 
delivering  more  than  150,000  patient  days  of  health  care. 

Data  Collection  Technicme (s)  Used:  Experiment  and  two 
questionnaires . 

Characteristics  of  Measuring  Instruments  Used:  A  quasi- 
experimental  research  design  was  employed  in  a  field  study, 
using  a  separate-sample  pretest-post-test  design  as 
described  by  Campbell  and  Stanley.  Job  perceptions  were 
measured  using  Job  Characteristics  Inventory  developed  by 
Sims,  Szilagyi,  and  Keller.  User  attitudes  were  measured 
using  five  point  Likert  type  instrument  developed  by  Schultz 
and  Slevin. 

Objective (a)  of  Measurement  Process:  Questionnaire  used  to 
collect  perceptual  and  attitudinal  data. 

Participants  in  Measurement  Process;  Nurses. 

Distribution  of  Measuring  Instruments:  Random  selection  of 
participating  nurses  with  258  completing  the  pretest 
questionnaire  and  218  completing  the  post-test 
questionnaire . 

Corrective  Action  taken  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Dropouts  from  experiment  were  studied  to  make 
suggestions  regarding  differences  between  participants  and 
non-participants . 

Method  Used  to  Analyze  Data :  Data  were  analyzed  using  the 
Statistical  Package  for  the  Social  Sciences  Version  H, 
Release  9.0. 

Issue  of  Validity  Addressed:  "The  design  (experiment)  did, 
however,  permit  an  assessment  of  the  effects  of  the  pretest 
on  post-test  scores,  a  common  threat  to  internal  validity  in 
experimental  design."  Authors  addressed  external  validity. 

Type  of  Validity  Claimed:  None. 


Source  of  Validity  Claimed:  Analysis  of  instrument 
measures . 

Type  of  Data  Recorded:  Ordinal. 

Secondairv  Source  (a)  Cited:  Schultz  and  Slevin  (1975) 
presented  the  original  instrviment,  however,  its  validity  was 
not  clearly  established. 

Sims,  Szilagyi,  and  Keller  (1976)  presented  analysis  of 
original  instrxunent  (Job  Characteristic  Inventory)  and 
claimed  to  have  demonstrated  construct  and  discriminant- 
convergent  validity.  The  source  of  the  claim  was  an 
analysis  of  the  instrxaments  measures . 
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Durand,  D.  E.,  R.  0.  Bennett  and  S,  Betty,  "What  Did 
Information  Technology  'Do'  to  Business  Communications? ; 
Two  Empirical  Studies,"  Information  &  Management,  Vol .  13, 
No.  3,  (October  1987),  pp.  111-117. 


Description  of  System  Studied:  Enable,  a  word  processing 
software  package. 

Data  Collection  Technique (s>  Used:  Experiment  using 
participant  observation. 

Characteristics  of  Measuring  instT~mw«»nts  Dsed:  Observations 
of  written  output  and  speed  of  reading  were  recorded  and 
analyzed. 

Objective (s)  of  Measurement  Process:  Observations  measured 
writing  style  and  content  and  speed  of  readina. 

Participants  in  Measurement  Process:  Students  in  an 
introductory  information  systems  course  were  observed  to 
measure  writing  style  and  content.  Business  professionals 
were  observed  to  measure  speed  of  reading. 

Distribution  of  Measuring  •  A  total  of  79 

students  participated.  It  was  assumed  that  the  observations 
were  made  in  the  student's  class.  Full  participation  of 
students  is  assiimed.  Distribution  to  business  professionals 
was  not  discussed,  nor  were  their  response  rates . 

Corrective  Action  taken  to  Account  for  Non-participants : 
Assvimed  full  participation  of  students.  Not  discussed  with 
business  professionals. 

Identification  of  Differences  between  Participants  and  Non- 
participanta :  Assumed  full  participation  of  students.  Not 
discussed  with  business  professionals. 

Method  Used  to  Analyze  Data:  Observations  (the  written 
output  of  students)  was  analyzed  using  Quintilian  Analysis, 
a  computer  program  specifically  designed  to  analyze  writing 
style  and  content.  Non-pareunetric  statistical  procedures 
(Mann-Whitney  U  for  ordinal  data)  and  parametric  statistics 
for  ratio  data.  Descriptive  statistics  were  used  to  analyze 
the  reading  speed  data . 

Issue  of  Validity  Addressed:  Not  discussed.  Internal  and 
external  validity  are  implied  in  findings. 
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Type  of  Validity  Claimed:  Internal  (implied)  and  External 
(implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Interval  and  frequency. 
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Gallupe,  R.  B.  and  G.  DeSanctis,  "Computer-Based  Support  for 
Group  Problem  Finding:  An  Experimental  Investigation," 
Management  Information  System  Quarterly,  Vol .  12  ,  No.  2, 
(June  1988),  pp.  277-296. 


Description  of  System  Studied:  DECAID,  a  group  decision 
support  system. 

Data  Collection  Technique (s)  Used:  Experiment  using  a  panel 
of  three  judges  to  compile  observations,  and  two 
questionnaires . 

Characteristics  of  Measuring  Instruments  Used:  Experiment 
was  of  pre  and  post  test  design.  The  panel  of  three  judges 
apparently  assumed  the  role  of  complete  observers; 
performing  document  analysis.  All  questionnaires  used  a 
seven  point  Likert  scale .  Interviews  were  conducted  in  a 
group  setting. 

Objective (s)  of  Measuramant  Process:  Observations  made  by 
the  three  judges  were  measures  of  decision  quality.  The 
questionnaires  and  interviews  measured  group  members' 
perceptions  of  group  decision  confidence  and  group 
satisfaction . 

Participants  in  Measurement  Process ;  Senior  undergraduate 
business  administration  students.  Random  assignment  of 
respondents  to  various  experiment  groups . 

Distribution  of  Measuring  Instruments:  A  total  of  84 
students  participated  in  the  study.  No  discussion  of 
response  rate  was  provided,  it  is  assumed  that  this  was  part 
of  a  class  assignment.  All  questionnaires  and  interviews 
were  administered  in  person. 

Corrective  Action  taken  to  Account  for  Non-participants: 

Assumed  full  participation. 

Identification  of  Differences  between  Pairticipants  and  Non¬ 
participants  :  Assumed  full  participation. 

Method  Used  to  Analyze  Data:  A  fixed-effect  two-way 
analysis  of  a  variance  model  was  applied  to  all  dependent 
measures . 
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Issue  of  Validity  Addressed:  "All  of  these  findings  must  be 
interpreted  with  caution,  given  the  limited  scope  and 
sampling  used  in  this  study.  As  a  controlled  experimental 
study  the  findings  cannot  necessarily  be  generalized  to  all 
levels  or  types  of  GDSS  software  or  to  actual  organizational 
use  of  GDSS  technology." 

Type  of  Validity  Claimed:  Internal  (implied) . 

Source  of  Validity  Claimed:  Not  discussed. 

Type  of  Data  Recorded:  Qualitative  and  ordinal. 
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Tait,  P.  and  I.  Vessey,  "The  Effect  of  User  Involvement  on 
System  Success :  A  Contingency  Approach"  Management 
Information  System  Quarterly,  (March  1988),  pp.  91-108. 


Description  of  System  Studied:  42  recently  implemented 
custom-built  information  systems,  within  Australian  firms. 

Data  Collection  Technique (s)  Used:  Questionnaires . 

Characteristics  of  Measuring  Instanaments  Used:  All 

questionnaires  were  either  off-the-shelf  or  employed 
techniques  used  in  previous  studies . 

Objective (s)  of  Measurement  Process:  Questionnaires 
administered  to  the  users  measured  success  of  the  system, 
user  involvement  in  design,  impact  of  the  system  on  the 
organization,  and  the  attitudes  of  the  users. 

Questionnaires  administered  to  the  designers  measured  the 
technical  complexity  of  the  system  and  the  resource 
constraints  on  the  development  of  the  system. 

Participants  in  Measurement  Procesa:  Users  and  designers  of 
the  information  systems. 

Distribution  of  Measuring  •  a  total  of  84  system 

users  and  designers  participated  in  the  study,  achieving  a 
71  percent  response  rate.  Questionnaires  were  sent  by  mail. 

Corrective  Action  taken  to  Account  for  Non-participants : 
Follow-up  phone  calls. 

Identification  of  Differences  between  Participants  and  Non¬ 
participants  :  Not  discussed. 

Method  Used  to  Analyze  Data:  Path  analysis. 

Issue  of  Validity  Addressed:  "Data  for  the  study  were 
collected  after  the  systems  were  implemented.  Subjects' 
responses  may  have  been  influenced  by  the  ultimate  success 
or  failure  of  the  systems .  The  timing  of  the  data 
collection  may  have  had  a  secondary  influence  on  the 
measurement  of  user  attitudes....  A  further  limitation  of 
the  study  is  the  use  of  instruments  that  are  not 
sufficiently  validated--the  perennial  problem  of  a  new 
discipline . " 

Type  of  Validity  Claimed:  None. 

Source  of  Validity  Claimed:  None. 
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Type  of  Data  Recorded:  Ordinal  and  possible  others. 


Secondary  Source (s)  Cited:  Questionnaire  was  comprised  of 
subsections,  using  instruments  developed  by  Wolek  (1975), 
Elizur  and  Guttman  (1976),  Ein-Dor  and  Segev  (1978),  Keen 
(1981) ,  McFarlan  (1981) ,  Ives,  Olson,  and  Baroudi  (1983) . 

Discussions  of  validity  were  presented  in  Elizur  and  Guttman 
(1976)  and  Ives  et  al .  (1983).  None  of  the  other 

discussions  significantly  siibstantiate  the  validity  of  this 
study . 
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Cronan,  T.  P.  and  D.  E.  Douglas,  "End-User  Training  and 
Computing  Effectiveness  in  Public  Agencies:  An  Empirical 
Study."  Journal  of  Management  Information  Systems,  Vol .  6, 
No.  4,  (Spring  1990),  pp.  21-39. 


Description  of  System  Studied:  End  user  computing  systems 
of  an  overall  public  agency  involved  in  financial  services. 

Data  Collection  Technicme (s)  Used:  Two  questionnaires  and 
analysis  of  existing  docviments  and  records . 

Characteristics  of  Measuring  instr^«n»nta  Used: 

Questionnaire  administered  to  users  completing  the  training 
program  used  a  five  point  scale  (Not  focus  of  study) . 
Separate  questionnaires  administered  to  end  users  and 
supervisors  use  a  five  point  type  scale  and  are  shortened 
versions  of  previously  used  instrviments  in  other  studies . 
Existing  documents  and  records  were  agency  statistics. 

Obiactiv(a)  of  Maasuremant  Process:  Questionnaires 
measured  training  prograun,  EUC  effectiveness,  and  end  user 
acceptance/satisfaction . 

Participants  in  Maasurmant  Procass:  Over  100  piablic 
personnel,  predominately  accountants.  Users  completing  the 
training  program  evaluated  the  training  program.  End  users 
and  supervisors  measured  end  user  computing  effectiveness 
and  end  user  acceptance/satisfaction. 

Distribution  of  Measuring  Instrmnants:  A  total  of  39 
responses  were  used  to  evaluate  the  training  prograun, 
distribution  and  response  rate  was  not  discussed. 

Corrective  Action  taJcan  to  Account  for  Non-participants : 

Not  discussed. 

Identification  of  Differences  between  Participants  and  Non- 
paxrticipants :  Not  discussed. 

Method  Used  to  Analyse  Data:  Paired  t-test  statistic,  Chi 
Square  test  statistic,  partied  t-test  statistic,  Wilcoxon 
signed  rank  test  statistic. 

Issue  of  Validity  Addressed:  Explained  types  of  validity 
and  presented  argximent  that  sufficient  degree  of  correlation 
exist  to  claim  reliadjility ,  convergent,  and  construct 
validity.  Advised  caution  should  be  exercised  in  complete 
generalizations  of  the  results. 

Type  of  Validity  Claimed:  Convergent  and  construct. 
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Source  of  Validity  Claimed:  Results  of  correlation 
analysis . 


Type  of  Data  Recorded:  Ordinal,  interval,  frequency,  and 
possible  others . 

Secondary  Source (s)  Cited:  Baroudi  and  Orlikowski  (1988) , 
Doll  and  Torkzadeh  (1988),  and  Ives,  Olson,  and  Baroudi 
(1983)  are  where  the  development  of  the  original  instruments 
were  presented.  Its  validity  was  discussed  by  all  the  above 
authors,  claiming  construct,  convergent-discriminant,  and 
criterion  validity.  The  source  of  the  claims  are  analyses 
of  measurements  from  the  instrument. 


APPEin)IX  c 


CONSOLIDATED  ANALYSIS  OF  STUDIES 

(Follows  on  succeeding  pages . ) 
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